Pose Estimation in Heavy Clutter using a Multi-Flash Camera

Transcription

1 2010 IEEE Internatonal Conference on Robotcs and Automaton Anchorage Conventon Dstrct May 3-8, 2010, Anchorage, Alaska, USA Pose Estmaton n Heavy Clutter usng a Mult-Flash Camera Mng-Yu Lu, Oncel Tuzel, Ashok Veeraraghavan, Rama Chellappa, Amt Agrawal, Haruhsa Okuda Unversty of Maryland Mtsubsh Electrc Research Labs Mtsubsh Electrc Corporaton {mngylu,rama}@umacs.umd.edu {oncel,veerarag,agrawal}@merl.com {Okuda.Haruhsa}@ct.MtsubshElectrc.co.jp Abstract We propose a novel soluton to object detecton, localzaton and pose estmaton wth applcatons n robot vson. The proposed method s especally applcable when the objects of nterest may not be rchly textured and are mmersed n heavy clutter. We show that a mult-flash camera (MFC) provdes accurate separaton of depth edges and texture edges n such scenes. Then, we reformulate the problem, as one of fndng matches between the depth edges obtaned n one or more MFC mages to the rendered depth edges that are computed offlne usng 3D CAD model of the objects. In order to facltate accurate matchng of these bnary depth edge maps, we ntroduce a novel cost functon that respects both the poston and the local orentaton of each edge pxel. Ths cost functon s sgnfcantly superor to tradtonal Chamfer cost and leads to accurate matchng even n heavly cluttered scenes where tradtonal methods are unrelable. We present a sub-lnear tme algorthm to compute the cost functon usng technques from 3D dstance transforms and ntegral mages. Fnally, we also propose a mult-vew based pose-refnement algorthm to mprove the estmated pose. We mplemented the algorthm on an ndustral robot arm and obtaned locaton and angular estmaton accuracy of the order of 1 mm and 2 respectvely for a varety of parts wth mnmal texture. I. INTRODUCTION Machne vson systems that robustly dentfy and locate objects have a multtude of applcatons rangng from ntellgent homes to automatc manufacturng. Although the populaton of robots has been growng fast, most robots work n restrcted and constraned envronments. For example, parts n assembly lnes have to be placed wth a fxed orentaton and poston for robots to grasp and manpulate them. It remans a great challenge to handle non-structured scenes contanng multple objects. Here, we propose a machne vson system that provdes (a) robust feature extracton (b) accurate pose estmaton and (c) fast operaton wth mnmal lag. A mult-flash camera [13](MFC) provdes accurate separaton of depth edges and texture edges n heavly cluttered envronments. We then reformulate the problem of object detecton and localzaton, as one of fndng matches between the depth edges obtaned n one or more MFC mages to the rendered (from CAD model) depth edges. We ntroduce a novel cost functon that respects both the poston and the local orentaton of each edge pxel. Ths cost functon s superor to Chamfer matchng cost and leads to accurate matchng even n cluttered scenes. Fnally, we also perform contnuous optmzaton to refne the estmated pose. A. Related Work Model-based pose estmaton usng 3D model to 2D mage correspondences can be found n [11][12][5]. Unfortunately Fg. 1: The robotc graspng platform. The MFC s mounted on the robotc arm and the objects are placed n a contaner. the 3D-2D pont correspondences are hard to obtan for ndustral parts due to ther textureless surfaces. The stuaton s partcularly severe when multple dentcal objects are placed together and overlap each other. Object contours provde rch nformaton about object denttes and ther poses. One fundamental problem n explotng ths cue s n matchng an exemplar contour to a newly observed contour, especally when the observed contour s partally occluded or located n a cluttered background. Besdes Chamfer matchng [3], varous contour matchng algorthms have been proposed n [7][2][10]. Although these methods perform better than Chamfer matchng for well segmented objects[3], ther performance n cluttered scenes tends to be nferor (see [17]). Edge orentaton can be used n order to mprove the Chamfer matchng n cluttered background as shown n [9][15][16]. Actve llumnaton patterns can greatly assst vson algorthms by extractng robust features n such challengng envronments. Examples of such technques nclude depth estmaton by projectng a structured llumnaton pattern [14]. In ths paper, we explot the MFC whch uses actve llumnaton n order to extract depth edges thereby removng the clutter from texture and shadow edges. In envronments where texture s ndeed a rch source of nformaton, we could stll use our algorthm wth canny edges nstead of depth edges obtaned va MFC. Snce the focus of ths paper s on ndustral envronments where texture s not an mportant vsual nformaton source, we restrct our attenton to depth edges from MFC for the rest of ths paper. Contrbutons: The techncal contrbutons are We show that a MFC provdes accurate extracton of depth edges even n heavy clutter. We ntroduce a novel cost functon that respects both the poston and the local orentaton of each edge pxel and show ts superorty over tradtonal edge based costs. We also develop a sub-lnear tme algorthm to compute the cost functon usng technques from 3D dstance /10/$ IEEE 2028

2 Fg. 2: System Overvew. transforms and ntegral mages. We propose a mult-vew pose-refnement algorthm n order to mprove the pose estmate. We mplemented the algorthm on an ndustral robot and obtaned locaton and angular estmaton accuraces of about 1 mm and 2 respectvely. II.SYSTEM OVERVIEW Physcal Layout and Calbraton: System desgn s shown n Fgure 1 where MFC s mounted to the robot arm. The MFC s calbrated (both nternal and external) usng a calbraton grd. Also, we performed the grpper-camera(handeye) calbraton so that the grpper can nteract and grasp objects. Ths gves us a complete calbrated framework. Algorthmc Layout: The algorthmc layout of the system s descrbed below and shown n Fgure 2. 1) Offlne Renderng of Database: For each object (a) render the 3D CAD model for every hypotheszed pose (b) compute the depth-edges and (c) ft lnes to the rendered MFC depth edges. 2) Feature Extracton: Capture 8 mages usng the 8 dfferent flashes of the MFC. These mages are then used to compute the depth edges n the scene. 3) Object Detecton and Pose Estmaton: Compute the matchng cost between each rendered pose of each object n the database and the edge map obtaned usng the MFC to perform coarse pose estmaton. Refne ths estmate further usng a mult-vew based poserefnement algorthm. 4) Graspng and Assembly: Once the 3D poses of the objects are accurately estmated, grasp the objects n order usng the grpper at the end of the robot arm and perform the requred assembly task. III.MULTI-FLASH CAMERA An MFC s an actve llumnaton based camera that contans 8 pont lght sources (LED s) arranged n a crcle around the camera as shown n Fgure 3. The MFC explots the change n the shadows caused by changes n llumnaton source postons n order to extract depth edges even for challengng objects such as textureless objects and mldly specular objects. Consder the MFC shown n Fgure 3. As the dfferent LED s around the camera flash, the poston of the shadows cast by the object change. Whle ponts on the object (such as P1) do not change ntensty as the flash moves around, ponts that are n the shadow of one of the flashes (such as P2) change ntensty sgnfcantly. Ths change n ntensty of the shadow pxels can be used to detect and extract vew dependent depth edges[13]. Fg. 3: Prncple of operaton of MFC. As the dfferent LED s around the camera flash, the shadows cast by the object change. Whle ponts on the object (such as P1) do not change ntensty as the flashes move around, ponts n the shadow of one of the flashes (such as P2) change ntensty. Ths s used to detect depth edges. Let us denote the mage captured (after ambent subtracton) durng the flashng tme of the th LED as I. The maxmum ntensty value at each pxel locaton among these ambent subtracted mages are found and used to construct the maxmum llumnaton mage I max (x,y)=max I (x,y). Next, we compute the rato mages of the ambent subtracted mages to the maxmum llumnaton mage, RI = I I max. Ideally, the rato value n the shadow regon (eg., pont P2) should be zero snce the contrbuton of the llumnaton from the ambent source has been removed. In contrast the rato values n other regons (eg., pont P1) should be close to one snce these regons are llumnated by all the flashes. Notce that the pont of transton between the pxels n the shadow regon and those not n the shadow regon s always a depth edge. For each rato mage, we apply a Sobel flter desgned to detect ths transton from shadow to non-shadow (0 to 1). IV.OBJECT DETECTION AND LOCALIZATION In ths secton, we present our algorthm for detecton and localzaton of objects n cluttered scenes usng depth edges acqured through MFC. Wthout loss of generalty, we descrbe the method as appled to a sngle object. However, ths assumpton s only for ease of presentaton, whle n realty the algorthm locates and estmates the pose of multple objects smultaneously. A. Database Generaton Gven the CAD model of the object, we generate a database of depth edge templates by smulatng MFC n software. In the smulaton, a vrtual camera havng the nternal parameters of the real MFC s placed at the orgn and ts optcal axs s algned wth the z-axs of the world coordnate system. Eght vrtual flashes are evenly placed on a crcle on x-y plane havng center at orgn and radus equal to the actual baselne between the camera and flashes. The CAD model of the object s then placed on the z-axs at a dstance t z from the vrtual camera. The vrtual flashes are swtched on one at a tme and eght renderngs of the object (ncludng cast shadows) are acqured. The depth edges n the scene are detected usng the procedure descrbed n Secton III. 2029

3 Fg. 4: Database generaton. We unformly sample the rotaton angles (θ x and θ y ) on the 2-sphere. The template database s generated by renderng the CAD model of the object wth respect to the sampled rotatons. An arbtrary 3D rotaton can be decomposed nto a sequence of three elemental rotatons about three orthogonal axes. We algn the frst of these axes to be the camera optcal axs and call the rotaton about ths axs as the n-plane rotaton (θ z ). The other two axes are on a plane perpendcular to the camera optcal axes and the rotaton about these two axes are called the out-of-plane rotaton(θ x and θ y ). Note that an n-plane rotaton results n an n-plane rotaton of the observed mages, whereas the effect of an out-of-plane rotaton depends on the 3D structure of the object. Due to ths dstncton, we only nclude out-of-plane rotatons of the object nto the database. We sample k out-of-plane rotatons (θ x and θ y ) unformly on the 2-sphere, S 2, as shown n Fgure 4 and generate the depth edge templates for each of these rotatons. B. Drectonal Chamfer Matchng Durng matchng, we search for the database template together wth ts optmal 2D Eucldean transformaton, s SE(2), whch algns the depth edges of the template to the query mage edges. A 2D Eucldean transformaton s represented wth three parameters, s =(θ z, t x, t y ), where t x and t y are the mage plane translatons along the x and y axes respectvely and θ z s the n-plane rotaton angle. Its acton on an mage pxel s gven as W(x;s)= ( cos(θz ) sn(θ z ) sn(θ z ) cos(θ z ) ) ( ) tx x +. (1) ty Chamfer matchng [1] s a popular technque to fnd the best algnment between two edge maps. Let U = {u } and V = {v j } be the sets of template and query mage edge maps respectvely. The Chamfer dstance between U and V s gven by the average of dstances between each pont u U and ts nearest edge n V d CM (U,V )= 1 n mn u v j. (2) v j V u U where n = U. The best algnment parameter ŝ SE(2) between the two edge maps s then gven by ŝ = arg mn s SE(2) d CM(W(U;s),V ). (3) Chamfer matchng becomes less relable n the presence of background clutter. To mprove robustness, several varants of Chamfer matchng were ntroduced by ncorporatng edge (a) (b) Fg. 5: Matchng costs per edge pont. (a) Shotten et al. [16]; (b) Drectonal Chamfer matchng. DCM jontly mnmzes locaton and orentaton errors whereas n [16] the locaton error s augmented wth the orentaton error of the nearest edge pont. orentaton nformaton nto the matchng cost. In [9], the template and query mage edges are quantzed nto dscrete orentaton channels and ndvdual matchng scores across channels are summed. Although ths method allevates the problem of cluttered scenes, the cost functon s very senstve to the number of orentaton channels and becomes dscontnuous n channel boundares. In [16], the Chamfer dstance s augmented wth an addtonal cost for orentaton msmatch whch s gven by the average dfference n orentatons between template edges and ther nearest edge ponts n the query mage. Instead of an explct formulaton of orentaton msmatch, we generalze the Chamfer dstance to ponts n R 3 for matchng drectonal edge pxels. Each edge pont x s augmented wth a drecton term φ(x) and the drectonal Chamfer matchng (DCM) score s gven by d DCM (U,V )= 1 n mn u v j + λ φ(u ) φ(v j ) (4) v j V u U where λ s a weghtng factor between locaton and orentaton terms. Note that the drectons φ(x) are computed modulo π, and the orentaton error gves the mnmum crcular dfference between the two drectons mn{ φ(x 1 ) φ(x 2 ), φ(x 1 ) φ(x 2 ) π }. (5) In Fgure 5, we present a comparson of the proposed cost functon wth [16]. In [16], the nearest pont n V s ntally located for a gven template pont u and the cost functon s augmented wth the dfference between ther orentatons, whereas the cost functon proposed n the paper jontly mnmzes the sum of locaton and orentaton error terms. It can be easly verfed that the proposed matchng cost s a pecewse smooth functon of both translaton t x, t y and rotaton θ z of the template edges. Therefore the matchng algorthm s more robust aganst clutter, mssng edges and small msalgnments. To our knowledge, the best computatonal complexty for the exstng Chamfer matchng algorthms s lnear n the number of template edge ponts, even wthout the drectonal term. In the followng secton, we present a sub-lnear tme algorthm for exact computaton of the 3D Chamfer matchng score (4). C. Search Optmzaton The search problem gven n (3), requres optmzaton over three parameters of planar Eucldean transformaton 2030

4 (a) (b) Fg. 6: Lnear representaton. (a) Edge mage. The mage contans edge ponts. (b) Lnear representaton of the edge mage. The mage contans 300 lne segments. (θ z, t x, t y ) for each of the k templates stored n the database. Gven a 640x480 query mage and a database of k = 300 edge templates, the brute-force search requres more than evaluatons of the cost functon n (4). We perform search optmzaton n two stages: (1) We present a sublnear tme algorthm for computng the matchng score; (2) We reduce the three-dmensonal search problem to onedmensonal queres by algnng the major lnes of template mages to the query mage. 1) Lnear Representaton: The edge map of a scene does not follow an unstructured bnary pattern. Instead, the object contours comply wth certan contnuty constrants whch can be retaned by concatenatng lne segments of varous lengths, orentatons and translatons. Here, we represent an edge mage wth a collecton of m-lne segments. Compared wth a set of ponts whch has cardnalty n, ts lnear representaton s more concse. It requres only O(m) memory to store an edge map where m << n. We use a varant of RANSAC [8] algorthm to compute the lnear representaton of an edge map. The algorthm ntally hypotheszes a varety of lnes by selectng a small subset of ponts and ther drectons. The support of a lne s gven by the set of ponts whch satsfy the lne equaton wthn a small resdual and form a contnuous structure. The lne segment wth the largest support s retaned and the procedure s terated wth the reduced set untl the support becomes smaller than a few ponts. The algorthm only retans ponts wth certan structure and support, therefore the nose s fltered. In addton, the drectons recovered usng the lne fttng procedure are more precse compared wth local operators such as mage gradents. An example of lnear representaton s gven n Fgure 6 where a set of ponts are modeled wth 300 lne segments. 2) Three-Dmensonal Dstance Transform: The matchng score gven n (4) requres fndng the mnmum cost match over locaton and orentaton terms for each template edge pont. Therefore the computatonal complexty of the bruteforce algorthm s quadratc n the number of template and query mage edge ponts. Here we present a threedmensonal dstance transform representaton (DT 3) to compute the matchng cost n lnear tme. Ths representaton s a three dmensonal mage tensor where the frst two dmensons are the locatons on the mage plane and the thrd dmenson s the quantzed edge orentaton. The orentatons are quantzed nto q dscrete channels ˆΦ = { ˆφ } evenly n [0 π) range. Each element of the tensor encodes the mnmum dstance to an edge pont n jont locaton and orentaton space: DT 3 V (x,φ(x)) = mn v j V x v j + λ ˆφ(x) ˆφ(v j ). (6) where ˆφ(x) s the nearest quantzaton level n orentaton space to φ(x) n ˆΦ. The DT 3 tensor can be computed n O(q) passes over the mage. Equaton (6) can be rewrtten as ( ) DT 3 V (x,φ(x)) = mn DT V { ˆφ } + λ ˆφ(x) ˆφ (7) ˆφ ˆΦ where DT V { ˆφ } s the two-dmensonal dstance transform of the edge ponts n V havng orentaton ˆφ. Intally we compute q two dmensonal dstance transforms DT V { ˆφ } usng the standard algorthm [6]. Subsequently, the DT 3 V tensor (7) s computed by solvng a second dynamc program over the orentaton costs, for each locaton separately. Usng the 3D dstance transform representaton DT 3 V the drectonal Chamfer matchng score of any template U can be computed n lnear tme va d DCM (U,V )= 1 n DT 3 V (u, ˆφ(u )). (8) u U 3) Dstance Transform Integral: Let L U = {l [s j,e j ]} j=1...m be the lnear representaton of template edge ponts U where s j and e j are the start and end locatons of the j-th lne respectvely. For ease of notaton, we sometmes refer to a lne wth only ts ndex l j = l [s j,e j ]. We assume that the lne segments only have drectons among the q dscrete channels ˆΦ, whch s enforced whle computng the lnear representaton. All the ponts on a lne segment are assocated wth the same orentaton whch s the drecton of the lne ˆφ(l j ). Hence the drectonal Chamfer matchng score (11) can be rearranged as d DCM (U,V )= 1 n l j L U u l j DT 3 V (u, ˆφ(l j )). (9) In ths formulaton, the -th orentaton channel of the DT 3 V tensor, DT 3 V (x, ˆφ ), s only evaluated for summng over the ponts of lne segments havng drecton ˆφ. Integral mages are ntermedate mage representatons used for fast calculaton of regon sums [18]. Here we present a tensor of ntegral dstance transform representaton (IDT3 V ) to evaluate the summaton of costs over any lne segment n O(1) operatons. For each orentaton channel, we compute the one-drectonal ntegraton along ˆφ (Fgure 7). Let x 0 be the ntersecton of an mage boundary wth the lne passng through x and havng drecton ˆφ. Each entry 2031

5 (a) (b) (c) (d) (e) Fg. 7: Computaton of the ntegral dstance transform tensor. (a) The nput edge map. (b) Edges are quantzed nto dscrete orentaton channels. (c) Two-dmensonal dstance transform of each orentaton channel. (d) The three dmensonal dstance transform DT 3 s updated based on the orentaton cost. (e) DT 3 tensor s ntegrated along the dscrete edge orentatons and ntegral dstance transform tensor, IDT 3, s computed. Fg. 8: One-dmensonal search. A template s rotated and translated such that one template lne segment s algned wth one lne segment n the query mage. The template s translated along the query lne segment and the drectonal Chamfer matchng cost s evaluated. of IDT3 V tensor s gven by IDT3 V (x, ˆφ )= DT 3 V (x j, ˆφ ). (10) x j l [x0,x ] The IDT3 V tensor can be n one pass over the DT 3 V tensor. Usng ths representaton, the drectonal Chamfer matchng score of any template U can be computed n O(m) operatons va d DCM (U,V )= 1 n l [s j,e j ] L U [IDT3 V (e j, ˆφ(l [s j,e j ])) IDT3 V (s j, ˆφ(l [s j,e j ]))]. (11) Snce m << n, the computatonal complexty of the matchng s sub-lnear n the number of template ponts n. The O(m) complexty s an upper bound on the number of computatons. For pose estmaton we would lke to retan only the best hypothess. We order the template lnes wth respect to ther support and start the summaton from the lnes wth the largest support. The hypothess s elmnated durng the summaton f the cost s larger than the current best hypothess. The supports of the lne segments show exponental decay, therefore for majorty of the hypotheses only a few arthmetc operatons are performed. 4) One-dmensonal Search: The search for the optmal pose over three parameters of planar Eucldean transformaton s computatonally ntensve to be practcal for real-tme applcatons. The lnear representaton provdes an effcent method to reduce the sze of the search space. The observaton s that, the template and query mage lne segments are near perfectly algned wth the true estmate of the template pose. In addton, the major lnes of the template and query mages are very relably detected durng the lne-fttng snce the procedure favors segments wth larger support. We order template and query lne segments based on ther support and retan only a few major lnes to gude the search. The template s ntally rotated and translated such that the template lne segment s algned wth the drecton of the query mage lne segment and ts end pont matches the start pont of the query segment as llustrated n the Fgure 8. The template s then translated along the query segment drecton and the cost functon s evaluated only at locatons where there s an overlap between the two segments. Ths procedure reduces the three-dmensonal search to one-dmensonal searches along only a few drectons. The search tme s nvarant to the sze of the mage and s only a functon of number of template and query mage lnes, and ther lengths. In almost all our experments, search along 5 template and 50 query mage lnes produces near dentcal results to bruteforce search and the optmal pose can be found under a few seconds for a database sze of k = 300 templates. V.POSE REFINEMENT The mnmum cost template together wth ts n-plane transformaton parameters (θ z, t x, t y ) provde a coarse estmate of the 3D object pose. Let θ x, θ y be the out-of-plane rotaton angles and t z be the dstance from the camera whch are used to render the template. We back project the n-plane translaton parameters to 3D usng the camera calbraton matrx K, and the ntal 3D pose of the object, p 0,sgvenby the three Euler angles (θ x,θ y,θ z ) and a 3D translaton vector (t x,t y,t z ) T. The 3D pose p can also be wrtten n matrx form M p = ( Rp t p 0 1 ) SE(3) (12) where R p s the 3x3 orthogonal matrx computed by a sequence of three rotatons around x y z axes R θz R θy R θx, and t p s the three-dmensonal translaton vector. The precson of the ntal pose estmaton s lmted by the dscrete set of out-of-plane rotatons ncluded nto the database. In ths secton, we present a contnuous optmzaton method to refne the pose estmaton. The proposed method s a combnaton of teratve closest pont (ICP) [19] and Gauss-Newton [4, pp.520] optmzaton algorthms. Three-dmensonal pose estmaton from a sngle vew s an ll-posed problem. To mnmze the uncertanty n pose estmaton, we use a two-vew approach, where the robot arm s moved to a second locaton and the scene maged wth MFC. The edge ponts detected n the two vews are gven by the sets V ( j) = {v ( j) }, j {1,2}. 2032

6 Let M ( j) SE(3), j {1,2} be the 3D rgd moton matrces determnng the locaton of the two cameras n world coordnate system and P =(K 0) be the 3x4 projecton matrx. The optmzaton algorthm mnmzes the sum of squared projecton error between the detected edge ponts v ( j), and the correspondng 3D ponts ũ ( j) n the 3D CAD model, smultaneously n both vews ε(p)= j ũ ( j) PM ( j) M p M ( j) 1 ũ ( j) v ( j) 2. (13) Note that, the projectons of 3D ponts ũ ( j) are expressed n homogeneous coordnates and n ths formulaton we assume that they have been converted to 2D coordnates. We fnd the 3D-2D pont correspondences va closest pont assgnment on the mage plane. We smulate the 2 camera setup and render the 3D CAD model wth respect to the current pose estmate p. Let U ( j) = {u ( j) }, j {1,2} be the sets of detected edge ponts n two synthetc vews and Ũ ( j) = {ũ ( j) } be the correspondng pont sets n the 3D CAD model. For each pont n U ( j) we search for the nearest pont n V ( j) wth respect to the drectonal matchng score arg mn v j V u v j + λ φ(u ) φ(v j ). (14) and establsh pont correspondences (ũ ( j),v ( j) ). The non-lnear least squares error functon gven n (13) s mnmzed usng the Gauss-Newton algorthm. Startng wth the ntal pose estmate p 0, we mprove the estmaton va the teratons p t+1 = p t + Δp. The update vector Δp s gven by the soluton of the normal equatons (J T ε J ε )Δp = J T ε ε, where ε s the N dmensonal vector of each of the summed error terms n (13), and J ε s the Nx6 Jacoban matrx of ε wth respect to p, evaluated at p t. The correspondence and mnmzaton problems are solved repeatedly untl convergence. The ntal pose estmate gven by the matchng algorthm s usually close to the true soluton, therefore n general 5 10 teratons suffce for convergence. VI.EXPERIMENT We performed an extensve evaluaton of the proposed system usng synthetc and real experments wth a robot arm. A. Experments on Synthetc Examples We quanttatvely evaluated the accuracy of the proposed system to detect and localze objects n hghly cluttered scenes on an extensve synthetc dataset. The synthetc dataset conssted of 6 objects of varyng complexty n ther 3D shape placed randomly one over the other to generate several cluttered scenes (Fgure 9). There were a total of 606 such synthetc mages that were rendered. The average occluson n the dataset was 15% whle the maxmum occluson was 25%. Moreover, n order to smulate mssng depth edges and mperfectons n the MFC, a small fracton (about 10 15%) of the depth edges was also removed. Detecton and Localzton: We compared the performance of the proposed cost functon to those of the Chamfer cost Det. Crcut Damond Ellpse T-Nut Knob Wheel Avg. Rate Breaker Toy Toy Ours [15] Chamfer TABLE I: Detecton Falure Rate comparson n hghly cluttered scene wth multple objects. Fg. 9: Examples of successful localzaton on the synthetc dataset. Frst column represents the sx dfferent target objects. functon and the orented Chamfer cost functon [15]. The detecton falure rate s shown n the Table I. The proposed matchng cost formulaton reduced the detecton falure rate of Chamfer matchng from 0.24 to It also reduced the error rate of competng state of art matchng formulaton of orented Chamfer matchng by half. We also observe that objects wth dscrmnatve shapes were easer for detecton such as the damond toy and the crcut breaker. On the contrary, the T-Nut object, whch has a smple shape, s relatvely hard for detecton snce false edges from clutter and other objects frequently confuse the optmzaton algorthm. Several examples of successful detectons for varous objects n challengng scenaros are shown n Fgure

7 1 Detecton Rate Degree of Occluson Fg. 10: Detecton rate versus percentage of occluson. Avg. t X t Y t Z θ X θ Y θ Z abs err. mm mm mm degree degree degree 1 Vew Vew TABLE II: Comparson of the average absolute pose estmaton error between the one-vew and two-vew approaches. Robustness to Occlusons: We further quanttatvely evaluated the robustness of the proposed cost functon aganst varyng degrees of occluson from no occluson to an average occluson of The results are presented n Fgure 10. We acheved greater than 99% detecton upto 5% occluson and about 85% detecton rate when one-fourth of the object s occluded. Two Vew Pose Estmaton We evaluated the pose estmaton algorthm usng a smlar synthetc settng. We randomly rendered a set of poses of varous objects. After a coarse pose estmate was computed, both the refnement scheme usng one vew and that of usng two vews were appled ndependently to further refne the estmate. The fnal estmates were compared to the ground truth and the results show that the two-vew approach outperformed the one-vew approach (Table II). 1mm corresponded to about 6.56 pxels on the mage plane ndcatng that the two-vew estmate was sub pxel accurate. B. Experments on Real Examples: Object Detecton and Pose Estmaton n Clutter: To quanttatvely evaluate the performance, we created several real test examples. Seven dfferent objects were lad one on top of another n a cluttered manner as shown n Fgure 11. We then performed object detecton, localzaton and pose estmaton on these real examples. The experment was repeated for several hundred trals and found that the detecton rate was about 95%. Shown n Fgure 11 are some typcal example trals of the real experment. In each mage, we render the slhouettes of the top detector outputs for three dfferent objects. We render the estmated depth edges of the objects over the actual captured mages n order to show the accuracy of the algorthm. Notce, that some of the parts have no texture whle others are mldly specular. Tradtonal mage edge (eg., canny) based methods usually fal n such challengng scenaros. The use of MFC allows us to robustly extract depth edges even n such challengng scenaros. Also notce, that snce the MFC feature s texture ndependent, the method works robustly for parts that have artfcal texture panted on them. Ths ndcates that the method can work n the presence of ol, grme and drt (whch are all common n ndustral envronments) all of whch add artfcal texture to the surface of objects. Statstcal Evaluaton: In order to statstcally evaluate the accuracy of the proposed system, we need a method for Fg. 11: Performance on real examples. The system detected and accurately estmated the pose for specular as well as textureless objects. Shown n these mages are the top detector outputs for three dfferent objects overlad on top of the orgnal mages. ndependently obtanng the 3D ground truth pose of the object. Snce there was no smple way of obtanng ths (especally for cluttered scenes), nstead we devsed a method to evaluate the consstency of pose estmate rrespectve of the vewpont of the camera. We placed an object n the scene. At each tme, the robot arm was commanded to perform a dfferent rotaton and translaton. From each of these vews, the MFC mage was obtaned and the pose estmate was obtaned n the camera coordnate frame. Snce the object s statc, the estmated pose of the object n the world coordnate system should be dentcal rrespectve of vewpont of the MFC. For each vew, the estmated pose of the object was transformed to the world coordnate frame usng knowledge of the poston and orentaton of the robot arm. The same experment was repeated for 7 dfferent objects wth 25 trals for each object wth the object n a dfferent pose n each tral. Durng each of these ndependent trals the robot arm was moved to 40 dfferent vewponts n order to evaluate the consstency of the pose estmate. The hstogram of the devatons (from the medan) of the pose estmate s shown n Fgure 12. The results demonstrate that the algorthm results n consstent estmates wth standard devaton of less than 0.5mm n the n-plane drectons (X,Y) and about 3 degrees n all three orentaton estmates. The standard devaton n estmate of Z ( Z-axs concdes wth the optcal axs of camera) s slghtly larger (about 1.2mm). C. Real Experments on Robot Arm We evaluated the algorthm on an ndustral robot arm as shown n Fgure 1. Several parts were thrown together n a bn to create cluttered scenes just as shown n Fgure 2034

8 estmaton accuraces of the order of 1 mm and 2 respectvely. REFERENCES Fg. 12: Hstograms of devatons from the pose estmates to ther medans n the real examples. 11. The grpper was made up of three vertcal steel pns each of 1mm dameter. The grpper of the robot arm was desgned to pck each of the objects by frst nsertng the three vertcal pns through a hole n the objects. Then the grpper opens the 3 pns thereby exertng horzontal force on the nsde edges of the hole. The hole n the objects was about 5mm to 8mm n dameter. Therefore, n order to successfully nsert the grpper nsde the hole (before lftng the object) the error n pose estmate should be less than about 1.5mm. When the pose estmate error s greater than about 1.5mm the pns are not nserted nto the hole and ths results n a falure to pck up the object. The proposed system s able to successfully gude the robot arm n the graspng task. We acheved a 0.95 graspng rate over several hundred trals. Among the 5% grasp falures a sgnfcant majorty (about 3%) were acually successful pose estmatons. But n these cases, the hole for graspng the target object was occluded by other objects. Hence, whle attemptng to pck these objects the grpper ht other objects and faled. We refer the readers to the supplemental materal for vdeos of the robot arm accomplshng ths task. In all these cases, the object detecton, localzaton and pose estmaton took an average of 6 seconds for an object n extremely cluttered envronments (on an Intel 2.66Ghz CPU wth 3GB memory). In envronments wth mnmal clutter the algorthm runs almost twce as fast snce there are much fewer edges on an average. On average the matchng task requres 2 seconds and pose refnement requres 2 seconds where the rest of the computaton tme s shared among depth edge extracton, thnnng and lne fttng. VII.CONCLUSION AND FUTURE WORK We presented a system for object detecton, localzaton and pose estmaton usng MFC. We formulated the problem as one of fndng matches between the depth edges obtaned n one or more MFC mages to the rendered depth edges that are computed offlne usng 3D CAD models of the objects. We ntroduced a novel cost functon that s sgnfcantly superor to tradtonal chamfer cost and developed mult-vew based pose estmaton and refnement algorthms. We mplemented the system on a robot arm and acheved locaton and angular [1] H. Barrow, J. Tenenbaum, R. Bolles, and H. Wolf. Parametrc correspondence and chamfer matchng: Two new technques for mage matchng. In Internatonal Jont Conference of Artfcal Intellgence, pages , [2] S. Belonge, J. Malk, and J. Puzcha. Shape matchng and object recognton usng shape contexts. IEEE Transactons on Pattern Analyss and Machne Intellgence, 24(4): , Aprl [3] G. Borgefors. Herarchcal chamfer matchng: A parametrc edge matchng algorthm. IEEE Transactons on Pattern Analyss and Machne Intellgence, 10(6): , November [4] S. Boyd and L. Vandenberghe. Convex Optmzaton. Cambrdge Unversty Press, March [5] D. DeMenthon and L. S. Davs. Model-based object pose n 25 lnes of code. In European Conference on Computer Vson, pages Sprnger-Verlag, [6] P. Felzenszwalb and D. Huttenlocher. Dstance transforms of sampled functons, [7] P. F. Felzenszwalb and J. D. Schwartz. Herarchcal matchng of deformable shapes. In IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, pages 1 8, June [8] M. A. Fschler and R. C. Bolles. Random sample consensus: a paradgm for model fttng wth applcatons to mage analyss and automated cartography. Graphcs and Image Processng, 24(6): , [9] D. M. Gavrla. Mult-feature herarchcal template matchng usng dstance transforms. In Internatonal Conference on Pattern Recognton, pages , [10] H. Lng and D. W. Jacobs. Shape classfcaton usng the nnerdstance. IEEE Transactons on Pattern Analyss and Machne Intellgence, 29(2): , February [11] D. G. Lowe. Three-dmensonal object recognton from sngle twodmensonal mages. Artfcal Intellgence, 31(3): , March [12] D. G. Lowe. Fttng parameterzed three-dmensonal models to mages. IEEE Transacton on Pattern Analyss and Machne Intellgence, 13(5): , May [13] R. Raskar, K.-H. Tan, R. Fers, J. Yu, and M. Turk. Non-photorealstc camera: depth edge detecton and stylzed renderng usng mult-flash magng. ACM Transactons on Graphcs, 23(3), [14] D. Scharsten and R. Szelsk. Hgh-accuracy stereo depth maps usng structured lght. IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, 1: , June [15] J. Shotton, A. Blake, and R. Cpolla. Contour-based learnng for object detecton. In IEEE Internatonal Conference on Computer Vson, volume 1, pages IEEE Computer Socety, October [16] J. Shotton, A. Blake, and R. Cpolla. Mult-scale categorcal object recognton usng contour fragments. IEEE Transacton on Pattern Analyss and Machne Intellgence, 30(7): , July [17] A. Thayananthan, B. Stenger, P. H. S. Torr, and R. Cpolla. Shape context and chamfer matchng n cluttered scenes. In IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, pages , June [18] P. Vola and M. Jones. Robust real-tme object detecton. Internatonal Journal of Computer Vson, 57(2): , [19] Z. Zhang. Iteratve pont matchng for regstraton of free-form curves and surfaces. Internatonal Journal of Computer Vson, 13(2): ,