Registration of Depth Image and Color Image Based on Harris-SIFT

Transcription

1 Registratio of epth mage ad Color mage Based o Harris-SFT Simi Zhao,Xiagmig Xu,Weilog Zheg,Jiawe Lig School of Electroic ad formatio Egieerig, South Chia Uiversity of Techology Guagzhou, Chia, zhao.sm@mail.scut.edu.c xmxu@scut.edu.c zheg.wl@mail.scut.edu.c jw.lig@mail.scut.edu.c Abstract the recet past, it has become a widely accepted ad applied method to recogize ad localize objects based o local poit features. However, the existig approaches are based o matchig color images. With the widely use of Kiect, the registratio betwee depth image ad color image becomes ecessary ad importat. To effectively match 3 depth images ad color images ad estimate the trasformatio homography, we preset a type of features, which combies Harris corer detector with SFT descriptor. the experimets, the correspodig feature poits betwee depth image ad color image ca be well located by the descriptor, which is ivariat uder blurrig, rotatio, shift, scalig ad moderate chages i viewpoit. Keywords-Harris-SFT; depth image; registratio. NTROUCTON Recet developmets i gamig techology, such as the Nitedo Wii, the SONY PlayStatio Move ad Microsoft Xbox360 Kiect, focus o robust motio trackig for compellig real-time iteractio, while the geometric accuracy ad appearace are of secodary importace. Kiect simultaeously captures 3 depth images ad color images at the rate of 30 frames per secod. Essetial beefits of this low-cost acquisitio device iclude ease of deploymet ad guarateed operability i atural eviromet. The user is ot required to wear ay physical markers or specialized makeup, ad the performace is ot adversely affected by itrusive light projectios or clumsy hardware cotraptios, either. However, these key advatages are achieved at the cost of a substatial degradatio i the data quality comparig to the state-ofthe-art performace capture systems based o markers or active lightig. As for further processig of depth image ad color image, the registratio betwee them becomes ecessary ad importat. We thus propose to register depth image ad color image to alleviate the problem of misaligmet. As we kow, the registratio ad aligmet of images take by multi-source cameras at differet spatial locatios ad orietatios i the same eviromet is a vital task to may applicatios i computer visio. Feature extractig ad feature matchig are two difficult tasks i automatic image registratio. Feature poits i scale space, which are affie-ivariat, provide a good way for feature extractig ad matchig. avid G. Lowe [1] proposed a ew feature extractig method amed scale-ivariace feature trasform (SFT), which have bee applied to feature extractig ad matchig widely ad proved to be steadily ivariat to image scalig ad rotatio []. Jia Gao [3] applied a ormalized Laplace detector to deal with image color iformatio ad perform real-time feature extractig ad matchig. Krystia Mikolajczyk [4, 5] proposed the Harris- Laplace detector ad proved that it has a better performace i repeatability, localizatio error ad scale chagig tha other detectors i scale space. Pedram Azad [6] combied Harris iterest poits ad the SFT descriptor for fast scaleivariat object recogitio betwee two color images. All these efforts have bee proved to work well i registratig itesity images. However, it is to be further ivestigated whether these features ca also work well o the registratio of depth image ad color image. this paper, we preset oe type of features, which combies Harris corer detector with SFT descriptor. order to achieve scale-ivariace i spite of omittig the scale space aalysis step of the SFT features, the features are explicitly computed at several predefied spatial scales. The mai orietatio of each feature poit is built i its eighborhood ad a feature descriptor is also costructed i the directio of its mai orietatio. The scale space projectio makes feature space resolutio-ivariat ad the feature s mai orietatio makes it rotatio-ivariat. additio, the feature matchig combies Euclidea distace with RANSAC to gai a better matchig. From the experimets, the correspodig feature poits betwee depth image ad color image ca be well located by the descriptor, which is ivariat uder blurrig, rotatio, shift, scale, ad moderate chages i viewpoit ad works perfectly i differet kids of trasformatio betwee images. 009, Wager et al. had developed a similar approach based o the same idea, combiig SFT descriptor ad Fers descriptor [7] together with FAST detector [8], as preseted i [9]. this paper, the origial SFT descriptor is combied with Harris corer detector, ad all parameters are derived from a thorough aalysis of the scale coverage of SFT descriptor. This paper is orgaized as follows. Sectio, the feature poits of the depth image ad the color image are detected. The details of feature poit detectio based o Harris corer detectio are itroduced. Sectio, we develop a SFT-like descriptor to describe the feature poits of the depth image ad the color image. Sectio V, the earest eighbor algorithm ad RANSAC algorithm are used to match the correspodig feature poits. Experimetal results ad aalysis are preseted i Sectio V.

2 Fially, we draw our coclusios i Sectio V.. FEATURE PONT ETECTON A crucial compoet of our registratio algorithm is how to fid the robust feature that ca properly describe the image. Stable feature poits ca be detected i both depth image ad color image with very diverse methodologies. We use a local feature poit descriptor to match the depth image ad the color image i this paper. Cosiderig the huge differece betwee 3 depth images ad color images, we should firstly preprocess both images respectively. The low resolutio ad high oise levels of the iput data are the primary challeges that we address i this paper. The image preprocessig is ecessary. As for depth image, we smooth them amog frames. We achieve depth pixel value as the media pixel values amog the successive 11 frames. practice, to ehace the cotrast of depth image ad fid more stable feature poits, we also apply the histogram equalizatio to depth image. As for color image, we blur it through the media filter with a 5 5 widow. order to localize feature poits, we use multi-scale Harris-Laplace corer detector. The Harris-Laplace detector relies heavily o both the Harris measure ad a Gaussia scale-space represetatio. The Harris-Laplace detector combies the traditioal Harris corer detector with the idea of a Gaussia scalespace represetatio i order to create a scale-ivariat detector. Harris-corer poits are good startig poits as they have bee prove to have good rotatioal ad illumiatio ivariace as well as the ability to idetify the iterestig poits of the image [10]. However, the poits are ot scale-ivariat ad thus the secod-momet matrix must be modified to reflect a scale-ivariat property. Let us deote M as the scale adapted secod-momet matrix used i the Harris-Laplace detector [11]. L ( ) L L ( ) x x y M =, ) = g( ) (1) L xl y ( ) L ( ) y Where g( ) is the Gaussia kerel of scale ad ω = (x, y). Similar to the Gaussia-scale space, L x is the Gaussia-smoothed image. The operator deotes covolutio. L ( ω, ) ad L ( ) are the derivatives i x y their respective directios applied to the smoothed image ad calculated usig a Gaussia kerel with scale. terms of our Gaussia scale-space framework, the parameter determies the curret scale at which the Harris corer poits are detected. Buildig upo this scale-adapted secod-momet matrix, the Harris-Laplace detector is a two-fold process: applyig the Harris corer detector at multiple scales ad automatically choosig the characteristic scale. The algorithm searches over a fixed umber of predefied scales. This set of scales is defied as: = k ( 1,,3...) () 0 = For each itegratio scale, chose from this set, the appropriate differetiatio scale is chose to be a costat factor of the itegratio scale: = s. our project we use k = 1. 5 ad s = Usig these scales, the iterest poits are detected through a Harris measure o the, ) matrix. The HrF, like the typical Harris measure, is defied as: HrF = det(, )) αtrace (, )) (3) Like the traditioal Harris detector, corer poits are those local maxima of the HrF that are above a specified threshold. For the depth image ad the color image, we use differet α ad the specified threshold. We the verify for each of the iitial poits whether Laplacia-Gauss respose attais a maximum at the scale of the poit. We reject poits for either the Laplacia attais o maximal or the respose is below a threshold. f the Harris feature makes the ormalized Laplace fuctio μ ) = L ( ) + L ( ) satisfies: ( xx yy ) > w, ) > w, ) ) ) > threshold The, this poit is chose as a Harris-Laplace feature poit. this way we obtai a set of characteristic poits with associated scales. For some poits the scale peak might ot correspod to the selected detectio scales of a image. Either these poits are rejected due to the lack of a maximum, or their locatio ad scale are ot very accurate. Thus it is ecessary to have a small scale iterval betwee two successive levels i order to fid the locatio ad scale of a iterest poit with high accuracy. With Normalized Laplace operator we fid the characteristic scale for each feature poit. f the feature poit has o characteristic scale, it's rejected. Similarly, the localizatio of the feature poits is calculated i sub-pixel accuracy, with Taylor approximatio.. FEATURE PONT ESCRPTOR SFT [1] is first preseted by avid G Lowe i As already stated, the SFT descriptor is a very robust ad reliable represetatio for the local eighborhood of a image poit. However, the scale-space aalysis requirig for the calculatio of the SFT feature poit positios is too slow for real-time applicatios. We combie the Harris L (4)

3 corer detector with the SFT descriptor i our project. order to omit the scale space aalysis step of the SFT features ad reduce the computatioal complexity, the features are computed at several predefied spatial scales explicitly. Through the previous process, we have detected the positios of the feature poits ad their characteristic scale. Similarly we ca use the SFT descriptor to describe the detected feature poits for further registratio. For each feature poit i those eighborhoods calculated derivatives ad with weighted histogram defied Mai Orietatio. Each key poit is assiged with cosistet orietatio based o local image properties so that the descriptor has a character of rotatio ivariace. This process is similar to Lowe's SFT descriptor. This step ca be described by two equatios below: m(x, y) = (L(x + 1, y) L(x 1, y)) + (L(x, y + 1) L(x, y 1)) (5) 1 L(x, y + 1) L(x, y 1) θ(x, y) = ta ( ) (6) L(x + 1, y) L(x 1, y) t's possible to ifluece this process: defie the size of the agle bis, if to take oly those feature poits that have oe very supported directio or to allow for the feature poits to have more tha oe mai orietatio, that are bigger tha some thresh. The highest peak i the histogram is detected ad the ay other local peaks with 80% of the highest peak value ca be assiged as the mai orietatio for optio. practice, we fid that the feature poits with more tha oe mai orietatio are preferable. After the mai orietatio is calculated, SFT-like descriptor is calculated for each feature poit with mai orietatio. ts differece from SFT is that whe calculatig derivatives ad their orietatios i eighborhoods of the feature poits, SFT-like descriptors are take i the directio of mai orietatio ear each 1 feature poit. The descriptio is take samples of them ad the step of samplig is take depedetly o characteristic scale, which is + 1. t is a fuctio that specifies step, as fuctio of scale. f the specified fuctio is costat 1, it's possible to take regular SFT descriptor. Also possible tuable optios are specified umber of widows for the descriptio ad size of agle bis. Each feature poit is described by a 4 4 regio aroud, each of which is further divided ito 4 4 sub-regios cosistig of 16 pixels. The i each of the 4 4 sub-regio, calculate the histograms with 8 orietatio bis. After accumulatio, the gradiet magitudes of the 4 4 regio to the orietatio histograms, we ca create a seed poit, a 8- demesioal vector. Therefore, a descriptor cotais elemets i total. This vector is the ormalized i order to ehace ivariace to chages i illumiatio. V. CORRESPONNG PONTS REGSTRATON As feature matchig requires high precisio, we combie Euclidea distace iitial match with RANSAC istead of oe sigle method. order to match the correspodig poits, we firstly use Euclidea distace betwee two feature vectors as the similar criteria of two key poits ad use the k-earest eighbor algorithm for registratio. By comparig the distace of the closest eighbor with that of the secodclosest eighbor we ca obtai a more effective method to achieve greater accuracy. Give a threshold, if the ratio of the distace betwee the closest eighbor ad the secodclosest eighbor is less tha the threshold, we the obtai a correct match. As for the filtered set of feature correspodeces resultig from the RANSAC algorithm, ow a full homography is estimated with a least squares approach Number of Poits matched dismatched Matchig Percetages ThreshFS Figure 1. Matched ad mismatched poits with differet ThreshFS ThreshFS Figure. Percetages of correct matches at differet ThreshFS V. EXPERMENTAL RESULTS A set of depth images ad color images are used to test

4 (a) (b) After usig the k-earest eighbor algorithm, we still have may mismatched poits due to the huge differece betwee depth image ad color image. So we use the RANSAC algorithm to elimiate outliers. The RANdom SAmple Cosesus (RANSAC) algorithm proposed by Fischler ad Bolles[13] is a geeral parameter estimatio approach desiged to cope with a large proportio of outliers i the iput data. Ulike covetioal samplig techiques which deped o eormous data to obtai a iitial solutio ad the proceed to prue outliers, RANSAC uses the smallest (c) set possible ad proceeds to elarge it with cosistet data poits, which could potetially lead to a wrog local miimum throughout the least squares approach for accurate homography estimatio. (d) (e) (f) (g) (h) (i) Figure 3. (a)(d)(g) feature poits of depth image (b)(e)(h) feature poits of color image (c)(f)(i) correspodig feature poits betwee depth image ad color image. The color images of first row ad secod row are captured by Kiect. the performace of our algorithm i this sectio. Except for the specified oes, all depth images are captured by Kiect ad their correspodig color images are captured by high defiitio camera. epth images are specifically processed for display. matchig features, a effective measure is obtaied by comparig the distace of the earest eighbour (NN) to that of the secod-earest eighbour (SNN) ad omittig the correspodeces i which the ratio NN/SNN is bigger tha ThreshFS. ThreshFS is the parameter to be determied. Figure 1 plots the results of matched ad mismatched poits with differet ThreshFS. Figure, the percetages of correct matches at differet ThreshFS are plotted. Throughout all experimets, the scale umber is 6 ad Harris thresh is set to 0.01 of the maximum Harris measure HrF for depth image ad for color image. Harris corer detectio, the miimal distace betwee two feature poits is pixels. The quality threshold was set to 0.05 for depth image ad for color image i order to produce eough feature poits. From the results, we ca fid that percetages of correct matches reach maximal whe ThreshFS is approximately 0.9. We firstly test the ability of our algorithm o highly symmetric ad similar objects. Figure 3 (a)~(c) show a sample of two earby views of a object (a umbrella). Each square shows oe matched feature, with the locatio, size, ad orietatio of the square idicatig the correspodig parameters for the Harris-SFT feature. As ca be see, there are about 9 matched poits. The algorithm has bee prove to work well i those highly symmetric ad similar images. Secodly, we would like to test the ability of our algorithm to symmetric ad more complex objects. Figure 3 (d)~(f) show a sample of aother target, a perso. The color image is captured by a rotated camera. As we ca see, whe the perso is stadig i frot

5 ACKNOWLEGMENT This work is supported by the Fudametal Research Fuds for the Cetral Uiversities, the Sciece ad Techology Plaig Project of Guagdog Provice uder the grad No.009B ad No.010A REFERENCES Figure 4. The result of the trasformatio of color image of the Kiect, the descriptor ca still match 11 correspodig feature poits. Experimetal results show that the algorithm ca effectively reduce the ifluece of camera rotatio. To evaluate the accuracy of our algorithm, we get the homography estimatio from the registratio ad show the result of trasformatio. Figure 3 (g)~(i) show a example of two differet viewpoits images ad the result of the registratio. Figure 4 shows the trasformatio from color image to the view of the depth image usig the homography estimatio. From the result, we ca see that the depth image captured by Kiect ad the color image captured by commo camera ca also be well matched. The trasformatio of the color image ca well match the origial depth image captured by Kiect, which shows the homography estimatio is accurate eough. The descriptor we use is ivariat uder blurrig, rotatio, shift, scalig ad moderate chages i viewpoit. V. CONCLUSONS this paper, we preset a type of features, which combies Harris corer detector with SFT descriptor. The practical applicability of our approach is verified by extesive experimetal evaluatio. the experimets, the descriptor is ivariat uder blurrig, rotatio, shift, scale ad moderate chages i viewpoit ad works perfectly i differet trasformatio betwee two types of images. The correspodig feature poits betwee depth image ad color image ca be well located. The robustess of the method is sufficiet for a rage of purposes. For example, with the registratio betwee depth image ad color image, we ca coveietly use the color image iformatio to improve the depth accuracy ad help three-dimesioal recostructio. Sice registratio ca also combie iformatio from aliged depth image ad color image ad severely reduce the oise i the data, it helps to tur depth cameras ad color cameras ito a viable tool for 3 shape scaig. [1] Lowe,.G., istictive mage Features from Scale-variat Keypoits, teratioal Joural of Computer Visio, vol. 60, o., pp , 004. [] Jua Geg, Ya Li ad Tao Chia., SFT based ris feature extractio ad matchig, Proceedigs of SPE, 007, Vol. 6753, 6753F. [3] Jia Gao, Xiha Huag ad Bo Liu, A quick scale-ivariat iterest poit detectig approach, Machie Visio ad Applicatios, 008. [4] Krystia Mikolajczyk ad Cordelia Schmid, dexig Based o Scale variat terest poits, Proceedigs of the 8th teratioal Coferece o Computer Visio, Vacouver, Caada, pp ,001. [5] Krystia Mikolajczyk ad Cordelia Schmid, A Scale ad Affie variat terest Poit etectors, teratioal Joural of Computer Viseio, vol. 60, o. 1,pp ,004. [6] Azad, P., Asfour, T. & illma, R., Combiig Harris terest Poits ad the SFT escriptor for Fast Scale-variat Object Recogitio, EEE/RSJ teratioal Coferece o telliget Robots ad Systems (ROS), St. Louis, USA, pp , 009. [7] Mustafa Ozuysal, Pascal Fua ad Vicet Lepetit, Fast Keypoit Recogitio i Te Lies of Code, EEE Coferece o Computer Visio ad Patter Recogitio, pp. 1-8, 007. [8] E. Roste ad T. rummod, Machie Learig for High-Speed Corer etectio, Europea Coferece o Computer Visio (ECCV), Graz, Austria, pp , 006. [9]. Wager, G. Reitmayr, A. Mulloi, T. rummod ad. Schmalstieg, Pose Trackig from Natural Features o Mobile Phoes, teratioal Symposium o Mixed ad Augmeted Reality (SMAR),Cambridge, UK, pp , 008. [10] Schmid, C., Mohr, R., ad Bauckhage, C. Evaluatio of iterest poit detectors, teratioal Joural of Computer Visio, vol. 37, o., pp , 000. [11] Mikolajczyk, K. ad Schmid, C. Scale & affie ivariat iterest poit detectors, teratioal Joural o Computer Visio, vol. 60, o. 1, pp.63-86,004. [1]. G. Lowe, Object Recogitio from Local Scale-variat Features, EEE teratioal Coferece o Computer Visio (CCV), Kerkyra, Greece, pp ,1999. [13] M.A. Fischler ad R.C. Bolles, Radom sample cosesus: A paradigm for model fittig with applicatios to image aalysis ad automated cartography, Commuicatios of the ACM, vol. 4, o. 6, pp , 1981.