Range Data Regstraton Usng Photometrc Features Joon Kyu Seo, Gregory C. Sharp, and Sang Wook Lee Dept. of Meda Technology, Sogang Unversty, Seoul, Korea Dept. of Radaton Oncology, Massachusetts General Hosptal, MA, USA samoosa@sogang.ac.kr, gcsharp@partners.org, slee@sogang.ac.kr Abstract Ths paper nvestgates the use of photometrc features for the par-wse regstraton of range mages. Many artfcal and natural objects exhbt abundant surface texture that may not be revealed n range data, and most structured lght and laser range sensors are capable of capturng ether grayscale or color photometrc ntensty n addton to range data. Nevertheless, the use of photometrc features has not been wdely nvestgated for range data regstraton, despte wdespread research nto local feature descrptors for object recognton n 2D photometrc mages. Ths paper addresses some of the problems that arse n usng photometrc features for range data regstraton, and presents a systematc method for ther use. Potentally useful photometrc features are detected on planar regons n 3D, and then reprojected to 2D to remove the perspectve dstorton. Then, a well-establshed 2D rotaton- and brghtness-nvarant mage feature descrptor s used for matchng. Range data algnment s performed usng a RANSAC algorthm, wth verfcaton performed n 3D. Expermental results demonstrate the effectveness of ths method. 1. Introducton Range sensors provde a fast and accurate way of measurng three-dmensonal (3D) surfaces, but the because of lmtaton n feld of vew, multple scans are requred for most objects or scenes. When the relatve poston and orentaton of each scan s unknown, varous regstraton algorthms may be used to algn the scans. Most regstraton methods try to optmze the dstances between overlappng surfaces, and some methods employ addtonal 3D features derved from the range ponts. However, few methods use photometrc features that can be extracted from the 2D photometrc mages, despte the fact that most commercal or homemade range sensors are capable of capturng regstered 2D photometrc and 3D range data. Ths paper dscusses a strategy whereby photometrc features can be used to smply and quckly regster range data n 3D. Tradtonal par-wse regstraton methods construct the correspondence sets by extractng salent features from both vews, and perform a search procedure to match the features. When relable features can be extracted and correspondences correctly establshed, the moton that best algns the two mages can be solved drectly. For low-profle 2.5D range mages, however, there may not be suffcently dstnctve local shape features to establsh correct correspondences. In other words, when a scene s rather plan, most features look the same. The dstnctveness further degrades when these features are processed to acheve rotatonal or affne nvarance. The teratve closest pont (ICP) algorthm and ts varants are a popular alternatve to the feature-based approach for range mage regstraton [1]. It s an teratve descent procedure that mnmzes the sum of the squared dstances between the ponts n the two vews. The correspondence problem s solved by assumng that the scene s approxmately algned wth the model and therefore that each scene pont corresponds wth ts closest model pont. However, the ICP method has the dsadvantage that a good ntal algnment s requred to acheve a satsfactory fnal regstraton. Wthout a good ntal algnment, many ntal condtons must be tred, and convergence to a global mnmum s unlkely. One soluton to ths problem s return to shape features as a gude to the regstraton process [2-7]. But shape features wll stll fal on scenes that are domnated by planar or smoothly varyng surfaces. The work presented n ths paper s based on the noton that features detected n the ntensty channel of the range data can provde a hgher level of dstnctveness than the shape features. Ths s an ntutvely smple dea, yet few methods are well known. One example s Johnson and Kang [8] who
descrbe an ICP algorthm that matches ponts usng ther color and range. However, ths approach stll suffers from correspondence errors, because by themselves, color values are not dstnctve. Roth used ponts of nterest extracted from the ntensty mages by the Harrs corner detector [9] [10]. Nonetheless, the method by Roth does not utlze the local mage nformaton for feature matchng, but reles only on the 3D range ponts assocated wth the detected feature ponts. On the other hand, Bendels et al. employed a well-establshed mage feature descrptor [13] for the regstraton of bas-relef-lke archaeologcal objects, but wthout any compensaton for 3D rotaton or projectve dstorton. [11] We ntroduce an approach that uses dstnctve photometrc features for the algnment of range mages. The chef focus of our work s on the modfcaton of the 2D mage features usng the 3D range nformaton for greater nvarance to projectve dstorton than can be acheved by exstng 2D mage feature descrptors. A fast, randomzed matchng algorthm s used for drect global regstraton wthout ntal algnment. For feature matchng, a 2D nvarant to rotaton s employed n our work to overcome the ambguty that remans even after the compensaton for projectve dstorton. There has been a consderable amount of research performed for generatng local features wth: rotaton nvarance [12], scale nvarance [13], and affne nvarance [14-15]. Our work benefts much from the recent advances n nvarant feature generaton research for object recognton. The rest of ths paper s organzed as follows. Secton 2 addresses the problems n usng the 2D photometrc features for range data regstraton and presents systematc solutons. Secton 3 dscusses the regstraton and algnment usng photometrc feature correspondences, and Secton 4 presents expermental results. Dscussons and summary are gven n Sectons 5 and 6, respectvely. 2. Photometrc feature matchng We propose a method that uses photometrc features to regster 3D range mages. Whle photometrc features are wdely used n matchng 2D mages, correctons are needed for geometrc and llumnaton varatons when used for 3D mages. 2.1 The problems wth photometrc features Photometrc features are a valuable source of nformaton for mage regstraton. However, two challenges must be addressed when computng feature values on three dmensonal range mages. The frst challenge s geometrc dstorton due to the change n camera pose, and the second challenge s photometrc dstorton due to change n llumnaton pose. We wll address those problems by concentratng on the photometrc features for planar or near-planar regons, where mage plane homography and ntensty normalzaton are suffcent. When the camera pose changes between vews, features become dstorted n the mage plane. For example, rotatng the camera about the target changes the projectve warpng as llustrated n Fgure 1. Our approach to ths problem s based on the feature rectfcaton n 3D space usng plane homography. scene correspondng features Features can also be dstorted by changes n llumnaton pose. For example, when the lght source s rotated about the target, the llumnaton change s not generally unform. For dffusely reflectng surfaces, the ntensty I(x,y) s related to the albedo ρ, the llumnaton L and the surface normal N(x,y) accordng to: I(x,y) = ρ L N(x,y). (1) Dscountng the effect of llumnaton change on a surface patch where N(x,y) vares substantally s not trval snce the accurate computaton N(x,y) on a hghly curvatured surface s dffcult. If N(x,y) s constant, on the other hand, the ntensty changes unformly wth changes n L, and normalzaton of feature wth respect to overall ntensty s suffcent. We enforce ths condton by lmtng our attenton to features that le n nearly planar surface patches. 2.2 Feature rectfcaton model Fgure 1. Photometrc feature matchng Feature rectfcaton s preprocessng step that allows us to accurately compute photometrc features from 3D range mages. It s desgned to address the dstortons n mage geometry and ntensty caused by changes n camera pose and llumnaton pose. The geometrc rectfcaton algorthm comprses the followng three stages: (1) Feature canddate detecton: The frst stage s to fnd feature canddates n the photometrc mages. In our mplementaton, we use the Harrs corner detector [9] over 5 5 or 7 7 wndows. All locatons that have a
detector response over a predefned threshold are selected as features. (2) Planarty check and surface orentaton estmaton: For each detected corner, the algorthm next checks the planarty of regon near the feature. If the feature s nearly coplanar, we wll use the feature for matchng. Otherwse, the feature s rejected. These checks are performed over a 15 15 regon usng the sngular value decomposton (SVD). Gven n ponts n 3D where X ~ [ ] T = X Y Z, we solve the SVD as: n T T UDV = X ~ X ~ X ~ X ~, (1) where X ~ m = n = 1 X ~ ( m )( m ) = 1 s the mean poston of the ponts. To check for coplanarty, we nspect the rato of the second largest sngular value wth the smallest sngular value. If ths rato s greater than a predefned threshold, the feature s used for matchng. For these features, we estmate the surface normal N as the column of V that corresponds to the smallest sngular value. Y y x Z ρ X Fgure 2. Surface and projecton geometry (3) Image warpng: Based on the surface orentaton, a wndow contanng each photometrc mage feature s warped to face the mage plane. Gven that the vewng drecton s known, and that we can recover the surface normal drecton for a gven pont, we can solve the dstorton of vewng drecton and scalng. The relatonshp between the captured and the warped mage features s plane homography. When a range sensor s calbrated, the 3 4 projecton matrx P n the followng equaton s known: x = P X = K[ R T]X, (2) where x = [ x y 1] T s the homogeneous coordnates n 2D mage, X = [ X Y Z 1] T s the homogeneous coordnates of 3D range ponts n each vew, and K, R and T are the ntrnsc, rotaton and translaton components of the projecton matrx P, respectvely. Fgure 2 llustrates the surface and camera projecton geometry. The frst step s to rotate the ponts so that N ê Z the ftted plane s algned wth the orgnal mage plane. Ths requres fndng a rotaton R S that algns the surface orentaton N, wth the drecton of the prncple axs of the camera ê Z,.e., the unt vector n the Z drecton. Ths s an underdetermned problem, and can be solved by addng an addtonal constrant. Let r be the unt vector n the drecton N ê Z. Then, we have the 3 3 rotaton matrx as: S [ ê r ê r][ N r N r] T R =. (3) Z Z We may take x as the projecton from a reference space as: x = K[ I 0]X. (4) Then, gven R, the mage coordnates for the rotated surface s gven as: x = K[ R S 0]X, (5) and the plane homography for mage warpng between x and x can be easly obtaned from: 1 x = H x, where H = KR S K. (6) Fnally, t remans to correct for the scale of the feature. If the orgnal focal length s f, and the dstance to the ftted plane s ρ, we have: scene ρ x = H x, where ρ = N T X ~. (7) m f N(x,y) rectfcaton N(x,y) model Fgure 3. Rectfcaton usng surface orentaton nformaton After rectfcaton, the correspondng features should appear dentcal n the mage plane up to rotaton about the surface normal as llustrated n Fgure 3. For the ambguty due to ths rotaton on the mage, a 2D feature nvarant to rotaton can be employed for matchng. 2.3 2D feature detecton and matchng After the rectfed mages are generated, we perform 2D local feature matchng usng the Scale Invarant Feature Transform (SIFT) algorthm, proposed by Lowe [13]. The SIFT algorthm was chosen because t ncorporates most of the recent advances n nvarant
feature generaton for recognton and t s one of the most effectve, fast and relable feature descrptors n our experence. It has several mportant propertes: (1) 2D-Rotaton nvarance: By algnng the drecton of the maxmum ntensty gradent wth the x-axs, we can acheve nvarance to the unknown 2D-rotaton of the rectfed mage. (2) Robustness to llumnaton changes: Because the lghtng pose changes for each vew, the ntensty vares wthn the planar regon. The effect of ntensty change s dscounted n SIFT n the process of extractng the edge of the mage. (3) Affne compensaton: By usng the surface orentaton to correct perspectve dstortons, the rectfed mages are senstve to nose of range data. Affne compensaton reduces the effect of mprecse surface normal computatons. 3. Regstraton When s s 3D pont n scene, and m s the correspondng 3D pont n model, we seek a rgd transformaton T that mnmzes the mean squared dfference between scene and transformed model: 2 ( ). (8) E = s T m When computng mean squared dfference, we have the flexblty of choosng a one to one matchng algorthm or a one to many matchng algorthm. One to one matchng s preferred when most of the correspondences are correct. However, one to many matchng can be useful when correspondence errors are common, due to repettve patterns or a low degree of overlap between scene and model. For such a multple matchng algorthm, we use a random samplng method lke RANSAC [16]. Frst, we compute a canddate transformaton from a randomly chosen set of correspondences, and then we verfy f the transformaton s reasonable or not. For a rgd 3D transformaton, a mnmum of three non-collnear pont correspondences s requred for a unque soluton. Wth more correspondences, the accuracy of the transformaton ncreases, but the probablty of msmatched ponts also ncreases. For our local photometrc features, we found that four features generally suffce. Verfcaton of the regstraton was accomplshed through a vew thresholdng technque [17]. After the model s algned wth the scene usng the photometrc feature correspondences, the 3D ponts n the model are thresholded nto nlers and outlers accordng to ther vsblty n the camera vewpont of the scene. The outlers, ponts that are not vsble n the scene, are dscarded. The nlers are used for verfcaton, by computng the mean squared dstance from nlers to ther matchng pont n the scene. Transformatons can be generated and verfed untl a suffcently good match between the surfaces s found. 4. Expermental Results We used a homemade structured lght range sensor, whch conssts of Epson EMP-7700 projector and Sony xc003 3CCD camera. It s calbrated wth a calbraton object pror to range scannng, and generates regstered range and color reflectance mages. Examples of the regstraton results generated by our method are shown n Fgs. 4-6. In Fgure 4, the model s rotated approxmately 50 degrees relatve to the scene, causng consderable dstorton of the photometrc features due to foreshortenng. Two vews and feature correspondences generated by the presented algorthm are shown n Fgures 4(a). Fgure 4(b) shows the matches generated usng the Lowe s reference mplementaton of SIFT [18]. Whle SIFT generates a many correct matches on the object to the rght, few matches are generated for the object on the left. Although our algorthm requres that surfaces used for matchng be locally planar, mldly curved surfaces can be used as well. Fgure 5 shows the regstraton results for a scene that s composed prmarly of curved surfaces. The regstraton succeeds for these knds of scenes because our rectfcaton procedure generates nearly dentcal photometrc patches, as shown n 5(d). Fgure 6 shows another set of results for a desktop scene. 5. Dscussons For many applcatons, we requre a regstraton algorthm that can algn scenes that contan mostly planar or smoothly varyng surfaces. For these scenes, algorthms that rely prmarly on geometry are faced wth the dffcult task of resolvng the ambguty n matchng planes aganst planes. The use of photometrc features n planar regons s a natural complement to shape features, and we antcpate future algorthms that blend these capabltes. The computatonal complexty of our algorthm s low compared wth other regstraton methods. Let each mage contan N ponts, F features be detected, nvarant mages be sze W, and let there be M matched ponts. Then, the complexty of fndng feature ponts s O(N), and the complexty of makng the descrptors s O(FW). The regstraton process s O(mM), where m s the number of random trals used n the RANSAC procedure.
If the probablty of a correct match s 50%, we requre 35 trals to correctly match 3 pars wth 99% probablty. The verfcaton stage s O(N) f correspondences are formed by projecton, and average case O(N log N) f formed usng closest pont. 6. Summary In ths paper, we propose new method of 3D range data regstraton usng 2D local photometrc features. The features are rectfed from 3D to 2D usng the surface normal, and algned usng randomzed matchng of 2D affne nvarants. Our proposed method s faster than most methods that rely on shape nformaton, and we have demonstrated strong results for scenes wth photometrc texture. (a) 7. Acknowledgements Ths work was supported n part by Grant No. R01-2002-000-004720 from the Basc Research program of Korea Scence and Engneerng (KOSEF), and n part by the Intellgent Robotcs Development Program, one of the 21st Century Fronter R&D Programs funded by the Mnstry of Commerce, Industry and Energy of Korea. (c) rectfcaton (e) (b) (d) Fgure 4. Plastc bottles: (a) two vews and feature correspondences from the presented method, (b) feature detecton and correspondences from Lowe s reference SIFT mplementaton, (c) algned range ponts (whte/black), (d) a pont-rendered novel vew, and (e) rectfed feature mages. (Note that the dsplayed mage sze s 50 50 for vewng clarty.)
(a) rectfcaton (b) (c) (d) Fgure 5. Shoe and bottle: (a) features and correspondences from the presented method, (b) algned range ponts (black/whte), (c) a pontrendered novel vew, and (d) rectfed feature mages. (b) (a) (c) Fgure 6. Desktop scene: (a) features and correspondences from the presented method, (b) algned range ponts (black/whte), and (c) a pontrendered novel vew. 8. References [1] P. J. Besl and N. D. McKay, A Method for Regstraton of 3-D Shapes, IEEE Transacton on Pattern Analyss and Machne Intellgence, 14(2):239-256, 1992. [2] G. C. Sharp, S. W. Lee and D. K. Wehe, ICP Regstraton usng nvarant features, IEEE Transacton on Pattern Analyss and Machne Intellgence, 24(1):90-102, 2002. [3] A. E. Johnson and M. Hebert. Surface matchng for object recognton n complex three-dmensonal scenes, Image and Vson Computng, 16(9):635-651, 1998. [4] C. Chua and R. Jarvs, 3d free form surface regstraton and object recognton, Internatonal Journal of Computer Vson, 17(1):77-99, 1996. [5] J. Feldmar and N. J. Ayache, Rgd, affne and locally affne regstraton of free-form surfaces, Internatonal Journal of Computer Vson, 18(2):99-119, 1996. [6] G. Soucy and F. P. Ferre, Surface recovery from range mages usng curvature and moton consstency, Computer Vson and Image Understandng, 65(1):1-18, 1997. [7] J. P. Thron, New feature ponts based on geometrc nvarants for 3d mage regstraton, Internatonal Journal of Computer Vson, 18(2):121-137, 1996. [8] A. E. Johnson and S. B. Kang, regstraton and Integraton of Textured 3D Data, Image and Vson Computng, 17(2):1345-147, 1999. [9] C. Harrs and M. Stephens, A combned corner and edge detector, Proc. Fourth Alvey Vson Conference, pp. 147-151, 1988. [10] G. ROTH, "Regsterng two overlappng range mages," Proc. Internatonal Conference on 3-D Dgtal Imagng and Modelng, pp. 191-200, 1999. [11] G. H. Bendels, P. Degener, R. Wahl, M. Koertgen, R. Klen, Image-Based Regstraton of 3D-Range Data Usng Feature Surface Elements, Proc. Internatonal Symposum on Vrtual Realty, Archaeology and Cultural Hertage, pp. 115-124, 2004. [12] Schmd, C., and Mohr, R., Local gray value nvarants for mage retreval, IEEE Transacton on Pattern Analyss and Machne Intellgence, 19(5):530-534, 1997. [13] D. G. Lowe. Object recognton from local scalenvarant features, Proc. Internatonal Conference on Computer Vson, pp. 1150-1157, 1999. [14] K. Mkolajczyk, C. Schmd An affne nvarant nterest pont detector, Proc. European Conference on Computer Vson pp. 128-142, 2002. [15] D. G. Lowe. Dstnctve Image Features from scale Invarant Keyponts, Internatonal Journal of Computer Vson 60(2):91-110, 2004. [16] M.A. Fschler and R.C. Bolles, Random sample consensus: A paradgm for model fttng wth applcatons to mage analyss and automated cartography, Communcatons of the ACM, 24(6):381-395, 1981. [17] T. Masuda and N. Yokoya. A robust method for regstraton and segmentaton of multple range mages, Computer Vson and Image Understandng, 61(3):295-307, 1995. [18] http://www.cs.ubc.ca/~lowe/keyponts/