RGBD Data Based Pose Estimation: Why Sensor Fusion?

Transcription

1 18h Inernaional Conference on Informaion Fusion Washingon, DC - July 6-9, 2015 RGBD Daa Based Pose Esimaion: Why Sensor Fusion? O. Serdar Gedik Deparmen of Compuer Engineering, Yildirim Beyazi Universiy, Ankara, Turkey gedik@ybu.edu.r A. Aydin Alaan, Senior Member, IEEE Deparmen of Elecrical and Elecronics Engineering, Middle Eas Technical Universiy, Ankara, Turkey alaan@eee.meu.edu.r Absrac Performing high accurae pose esimaion has been an aracive research area in he field of compuer vision; hence, here are a pleny of algorihms proposed for his purpose. Saring wih RGB or gray scale image daa, mehods uilizing daa from 3D sensors, such as Time of Fligh (TOF) or laser range finder, and laer hose based on RGBD daa have emerged chronologically. Algorihms ha exploi image daa mainly rely on minimizaion of image plane error, i.e. he reprojecion error. On he oher hand, mehods uilizing 3D measuremens from deph sensors esimae objec pose in order o minimize he Euclidean disance beween hese measuremens. However, alhough errors in associaed domains can be minimized effecively by such mehods, he resulan pose esimaes may no be of sufficien accuracy, when he dynamics of he objec moion is ignored. A his poin, he proposed 3D rigid pose esimaion algorihm fuses measuremens from vision (RGB) and deph sensors in a probabilisic manner using Exended Kalman Filer (EKF). I is shown ha such a procedure increases pose esimaion performance significanly compared o single sensor approaches. I. INTRODUCTION Many compuer vision relaed problems can be solved effecively wih he aid of 3D objec racking. In roboic applicaions, knowing he exac meric locaion of eniies relaive o each oher enables user defined acions o be performed by auomaic sysems. For insance, esimaion of he relaive orienaion beween Unmanned Aerial Vehicle (UAV) and anker reference frames gains auonomous refueling capabiliy o UAVs. Considering moneary as well as human healh issues, highly accurae posiion esimaion is criical for he reliabiliy of such sysems. As a miliary sysem, on May 2012, DARPA and Norhrop Grumman Corp. announced compleion of an auonomous refueling sysem beween wo high aliude UAVs [1]. Augmened Realiy (AR) is also an aracive applicaion of pose esimaion. AR simply aims inserion of arificial objecs o a real scene observed by a camera. To his aim, he relaive orienaion beween capuring camera and he scene should be discovered. The challenge in such applicaions is o obain consisen pose esimaes a consecuive ime insans in order o avoid jier. In a pracical sysem an auomobile company aims o assis echnical saff during car mainenance by inserion of synheic informaion ono he observed moor video [2]. Fig. 1: 3D scene map [4] In some applicaions, in which he objec is represened as a poin cloud model (PCM), esimaion of orienaion beween camera and objec reference frames enables 3D mapping of he objec. For insance, he map shown in Figure 1 is obained by racking he objec for more han 1000 frames and for such cases ha may suffer from error accumulaion over ime, pose esimaion accuracy becomes criical in order o generae pleasan 3D maps. There are commercial soluions ha aim o bring such a echnology ino our homes [3]. Recen advances in sensor echnology, creaed sensors, such as SwissRanger Sr-4000 [5], Microsof Kinec [6] and Asus Xion [7], ha can capure high resoluion and high frame rae 3D as well as 2D visual daa. Among hese sensors, Kinec and Xion are widely used among compuer vision researchers, since hey are affordable as well as providing ime synchronous RGB and 3D daa. Figure 2 shows ypical regisered RGB and deph images ha are capured by Kinec. In his paper, our aim is o increase he accuracy of 3D pose esimaion by simulaneous uilizaion of visual and deph daa provided by he sensors. Figure 3 illusraes he objec and sensor reference frames. Following variables define he overall sysem: X oi : 3D coordinaes of i h objec poin wih respec o he objec reference frame, ISIF 2129

2 (a) RGB image (b) Deph image Fig. 2: Typical RGB and deph images capured by Kinec [8]. X odi Y odi Z odi Fig. 3: Objec and sensor reference frames. : 3D coordinaes of i h objec poin measured by he deph camera wih respec o he deph camera reference frame, [ ] xovi : 2D pixel coordinaes of i y h objec poin measured by he vision sensor, ovi R do = R(ρ do,θ do,φ do ) : Roaion marix beween he objec and he deph camera reference frames defined by angles ρ do, θ do and φ do in x, y and z direcions, respecively, do = [ xdo, ydo, zdo ] T : Translaion vecor beween he objec and he deph camera reference frames in x, y and z direcions, respecively, R vd = R(ρ vd,θ vd,φ vd ) : Roaion marix beween he deph and he vision sensor reference frames defined by angles ρ vd, θ vd, and φ vd, in x, y and z direcions, respecively, vd = [ xvd, yvd, zvd ] T : Translaion vecor beween he deph and he vision sensor reference frames in x, y and z direcions, respecively. Problem is defined as recovering he ransformaion beween objec and deph camera reference frames, i.e. R do and do. We assume ha exernal calibraion beween vision and deph sensors, i.e. R vd and vd, is already esimaed and remains fixed in ime. Please refer o [9] for a discussion on exernal calibraion of he sensors. 3D objec coordinaes and measuremens of he deph sensor are relaed as follows (Time index is omied for he sake of simpliciy): X odi Y odi Z odi = R do X oi + do (1) Noe ha once 3D-3D correspondences beween objec coordinaes and deph sensor measuremens are known or esimaed, i is possible o recover desired ransformaion parameers. Mehods uilizing deph only measuremens are examined in Secion II. Similarly, ignoring lens disorions [10],vision sensor measuremens can be relaed o he objec coordinaes: α i x ovi 1 = K v R vd R do X oi + do + vd (2) where α i is he scale facor and K v is he inernal calibraion marix of he vision sensor. K v is relaed wih he physical consrucion of he vision sensor and according o pin-hole camera model i can be wrien as follows [10]: K v = f x s p x p y (3) 1 where f x and f y represen focal lenghs, p x and p y represen principal poin coordinaes (all in pixels) and s is he skew parameer. Rewriing (2), we obain he following relaion for vision sensor measuremens: where α i x ovi 1 f y = K v R vo X o i + vo (4) R vo = R vd R do (5) vo = R vd do + vd Similar o (1), once 2D-3D correspondences are esablished, i is possible o recover K v, R vo and vo, and hence, he required ransformaion [R do, do ]. In he lieraure, he esimaion of K v and [R vo, vo ] from correspondences are referred as inernal camera calibraion and pose esimaion, respecively. Alhough hese wo problems are coupled and can be solved simulaneously, hroughou his paper, we assume ha K v is calculaed offline and remains fixed. The algorihms proposed for pose esimaion using 2D-3D correspondences are reviewed in Secion II. Alhough i is possible o esimae he objec pose using single sensor approaches, in his paper we show ha by fusing boh sensor measuremens in a probabilisic manner, i is possible o increase he accuracy of pose esimaion. To his aim, EKF is adoped as he probabilisic framework due o is abiliy o model he dynamics of he objec moion. 2130

3 II. RELATED WORK For pose esimaion using 2D-3D correspondences, many soluions have been proposed. Similar o he classificaion menioned in [11], in a more general manner, hese algorihms can be grouped ino wo caegories as follows: 1) Algorihms direcly uilizing 2D-3D correspondences o minimize errors, such as reprojecion error, objec space lineariy error, ec. 2) Algorihms relying on he esimaion of objec coordinaes wih respec o he camera reference frame. In one of he earlies approaches falling ino he firs caegory, a mehod named POSIT (Pose from Orhography and Scaling wih Ieraions) is proposed [12]. In POSIT, by updaing he iniial scale facor hrough he ieraions, he perspecive projecion is esimaed by scaled orhographic projecion, unil convergence. The exended version called SOFTPOSIT [13] solves he assignmen, i.e. maching beween 2D image and 3D objec coordinaes, and pose esimaion problems simulaneously. The assignmen problem is solved by he sofassign algorihm of [14]. SOFTPOSIT algorihm is observed o be sensiive o iniial condiions; hence, may diverge if no properly iniialized. The Direc Linear Transformaion (DLT) [15] mehod esimaes he 4 3 ransformaion marix beween 3D objec coordinaes and 2D image coordinaes direcly wihou forcing he rigid body ransformaion model and hen he ransformaion marix is decomposed ino inernal camera, roaion and ranslaion marices using RQ decomposiion. Since his simple mehod is no accurae, i is generally uilized as an iniial esimae in ieraive approaches. However, due o inheren sensor noise, i is no rivial o decompose projecion marix in order o ge an orhogonal roaion marix and error is inroduced during his decomposiion. Wih his moivaion, he algorihm proposed by he auhors in [16] uilizes orhogonal ieraions (OI) mehod, which guaranees an orhogonal roaion marix R hrough ieraions and decreases he objec reference frame space error unless a soluion is reached. The image poins are uilized as hypohesized scene poins in order o obain an iniial esimae and his iniializaion is saed o resul wih a pose esimaion beer han a weak-perspecive iniializaion. Finally, he auhor of [17] proposes an efficien linear soluion for he exerior orienaion esimaion problem. Orhogonal decomposiion is firs used o isolae he unknown dephs of feaure poins wih respec o he camera reference frame. This approach allows he problem o be reduced o an absolue orienaion wih scale problem, which is solved using he Singular Value Decomposion (SVD). The algorihms in he second caegory sem from he Perspecive-n-Poins (PnP) approach (specifically P3P approach) developed by Gruner in 1841 [18], which is sill a highly popular mehod in pose esimaion lieraure. The mehod uilizes he relaive disances beween he coordinaes of feaures in objec reference frame and provides a closed form soluion o he corresponding 3D coordinaes in he camera reference frame. Finally, he wo reference frames are relaed by solving he absolue orienaion problem. In [19], possible P3P soluions are analyzed. The P3P mehod is quie sensiive o noise, since i depends on he soluion of higher order polynomials for deerminaion of he camera coordinaes. Moreover, P3P algorihm may yield up o 4 real soluions. Hence, in [20], he auhors propose a linear mehod, which is based on solving many P3P equaions from n (n > 3) poins, using SVD. However, his approach is no sable when here are ouliers and a posiive soluion is no guaraneed. Therefore, he auhors of [21] propose a PnP soluion uilizing Gauss-Newon ieraions based on he iniial esimaes of [20]. The ieraed manner, however, may decrease efficiency. An efficien non-ieraive PnP algorihm, which is based on expressing he 3D feaure poins as a weighed sum of four virual conrol poins and esimaing he coordinaes of hese conrol poins in he camera reference frame, is proposed in [22]. The mehod is saed o be efficien wih O(n) complexiy. Moreover, alhough heir accuracies are generally lower han menioned mehods, here are numerous algorihms uilizing paern recogniion echniques [23] in he lieraure of pose esimaion. For insance, he head pose esimaion algorihm proposed in [24] uilizes specral regression discriminan analysis wih auomaic regularizaion parameer esimaion. The mehod is claimed o yield promising pose esimaion resuls. There are also muli-view approaches proposed for pose esimaion. In [25], daa fusion is performed by back-projecions from single images of he muli-view se ono he esimaed 3D model. Then, he model pan angle is esimaed by uilizing a paricle filer. Furhermore, in [26], a neural nework-based muli-view pose esimaion scheme is proposed. The algorihms uilizing pure range daa are based on he regisraion of 3D poin clouds, especially when he 3D-3D correspondences are no known. For insance in [27], in offline phase, a riangular mesh model of he objec is formed using range sensors. In he online sage, he mesh model of he objec is aligned wih he capured range daa using Ieraive Closes Poin (ICP) approach. Proposed in [28], ICP algorihm firs maches he closes poins beween wo poin ses and calculaes pose based on his associaion. Then aligns poins using his pose esimae and in an ieraive manner coninues o mach neares poins and esimae pose. For pose esimaion, quaernion-based approach, which represens roaion marix using a 4 1 uni norm vecor, is used. The quaernions are calculaed by minimizing a mean square objecive funcion ha calculaes he difference beween original and ransformed 3D poins. The auhors of [27] modify ICP so ha i can be implemened in real-ime. The main drawback of ICP algorihm is is sensiiviy o iniial objec pose. Since ICP requires a good iniial esimae o converge, many algorihms exploi ICP afer a coarse regisraion. In [29], using he limis of objec velociy and he sensor frame rae, he inerframe ransformaion space is reduced considerable and he pose space is quanized; hence he problem of pose esimaion is convered o a classificaion problem. Following discree 2131

4 pose classificaion sep, ICP algorihm is uilized o fine-une pose esimaes wih a few ieraions. Furhermore, uilizing parallel processing on GPUs, in [30] an ICP based 3D racking algorihm is proposed. The sysem racks all pixels in a 640x480 image and he performance is quie saisfacory due o dense racking. However, he huge compuaional burden makes i impossible o run he algorihm on convenional deskop plaforms. In order o perform 3D-3D regisraion a descripor based approach is proposed in [31]. Firs of all, for each daa poin, a descripor based on local geomery is compued. Then disincive feaures are seleced among daa poins based on he uniqueness of heir descripors. Then, a disance marix soring he descripor disances beween model and daa poins is formed. Opimal se of correspondences, which brings he ses o a coarse alignmen, is esablished using he branchand-bound algorihm. The pose is refined furher using ICP mehod. However, in he case of noisy measuremens, a descripor-based mehod may no yield saisfacory resuls. One of he main drawbacks of working wih range daa is he inheren sensor noise and low resoluion. Hence, in order o overcome such limiaions, many researches propose o uilize range sensors and vision sensors ogeher. In [32], he 2D-3D correspondences beween range and vision daa are uilized for RANSAC [33] and LM [34] based pose esimaion. Sensor fusion is only uilized o calculae covariance marices of 3D measuremens and pose racking is insananeous. To handle such drawbacks, in [35], firs he objec is ransformed using he iniial sae esimae of EKF. Then, he proposed ariculaed ICP is used o align poin clouds. Finally, measuremen updae is performed o correc he pose esimae of he ariculaed ICP. Moreover, he pose esimaes of EKF and ICP based mehods correcs each oher a each ieraion. Similarly, he auhors of [36] maximize phoo consisency by linearizing he cos funcion o regiser all pixels of consecuive frames. The auhors of [37] propose a probabilisic opimizaion framework in order o regiser RGBD images and rack pose. The join shape and color disribuions are represened as a ree srucure, where each node sores saisics on he join spaial and color disribuions of he poins wihin is volume. The graphs a differen ime insans are associaed by finding he ransformaion (represened by uni quaernions and a ranslaion vecor) ha maximizes heir maching likelihood. Alhough mehod is accurae, i is compuaionally involved. Alhough no poined by he above algorihms, he moion of a free moving objec can be modeled using a dynamic sysem framework, such ha he curren pose of he objec is consrained wih he previous ime insan pose and he underlying moion model. This formulaion enables ime consisen racking resuls; hence, reduces jier. By his moivaion, he algorihm of [35] performs a weighed combinaion of objec pose imposed by moion model and ha esimaed using an ICP varian operaing on RGBD images. However, in his approach a higher level fusion of esimaed pose parameers is proposed and underlying noise saisics inheren in he measuremens of vision and deph sensors is ignored. III. PROBABILISTIC FUSION OF RGBD DATA As illusraed in Figure 3, he join uilizaion of vision and deph sensors enables wo ses of measuremens for each objec poin i, namely [X odi,y odi,z odi ] T from deph sensor and [x ovi, ] T from vision sensor. In order o increase he accuracy of 3D pose esimaion, one should devise an algorihm exploiing hese wo ypes of daa as much as possible. Researchers involved in probabilisic roboics research mainly deal wih solving similar problems. The robo needs o make an esimae of he sae using measuremens acquired by is sensors. The erm sae sands for any propery of he robo or he environmen of ineres, such as he velociy or posiion of he robo and he locaions of feaures around he robo [38]. The saes change in ime according o a sysem model. For insance, if you apply his much power o he wheel moors, he velociy of he robo changes ha much. These saes canno be direcly observed by he robo bu hrough some sensor measuremens, such as he readings of an odomeer sensor or he images capured by a vision sensor. Similarly, hese measuremens are mahemaically relaed wih he saes. To sum up, wo relaions govern he overall sae esimaion procedure: 1) Sae Updae Equaions: Wha is he sae x a ime, if a sequence of conrol inpus u 1: are applied, a sequence of saes x 0: 1 are resuled and a sequence of measuremens z 1: 1 are observed? 2) Measuremen Equaions: Wha is he measuremen z a ime, if a sequence of conrol inpus u 1: are applied, a sequence of saes x 0: are resuled and a sequence of measuremens z 1: 1 are observed? Saes and measuremens evolve in ime according o probabilisic laws [38]. Sae and measuremen equaions are governed by following probabiliy disribuions, respecively: p(x x 0: 1,z 1: 1,u 1: ) p(z x 0:,z 1: 1,u 1: ) Alhough he robo canno direcly observe saes, i can have a belief regarding he saes hrough measuremen and conrol inpu sequences: (6) bel(x ) = p(x z 1:,u 1: ) (7) Defined formally, belief is he robo s inernal knowledge of he sae. A wo sep approach gives he belief. Firs, before uilizing he curren measuremen z, a predicion bel(x ) is made: bel(x ) = p(x z 1: 1,u 1: ) (8) Then afer incorporaingz,bel(x ) is obained frombel(x ) by measuremen updae. Bayes filer algorihm is he mos general mehod for calculaing beliefs [38]. In a Markov process, he predicion and measuremen updae seps of he algorihm are respecively as follows: 2132

5 bel(x ) = p(x x 1,u )bel(x 1 )dx bel(x ) = ηp(z x )bel(x ) where η is he normalizing scalar. Please noe ha, once he dynamical sysem model, defined by sae updae and measuremen equaions, is known, i is possible o obain he probabiliies p(x x 1,u ) and p(z x ); hence, he belief, using Bayes filer approach. For he 3D pose esimaion problem of concern, in which he objec makes free movemens and associaed measuremens are acquired by sensors, Bayes algorihm sands as a powerful ool o esimae pose, since i makes ime consisen esimaions in a well-defined probabilisic framework. The mahemaical derivaions of Bayes algorihm are no in he scope of his manuscrip. Refer o [38] for furher discussions. Kalman Filer (KF) is proposed for he soluion of Bayes algorihm in linear sysems, for which he sae ransiion (p(x x 1,u )) and measuremen (p(z x )) probabiliies are linear in erms of heir argumens wih addiive Gaussian noise [39]. However, as inroduced in Secion I, geomeric relaions involved in model-based 3D racking have non-linear characerisics. Therefore, KF formulaion does no direcly sui our needs. A his poin, Exended Kalman Filer (EKF) proposed for non-linear sysems comes as a soluion, for which sae updae and measuremen equaions are governed by funcions g and h, respecively: x = g(u,x 1 )+ǫ (10) z = h(x )+ε where ǫ and ε are zero mean random Gaussian vecors sanding for randomness in sae ransiion and measuremens wih covariance marices R and Q respecively. Consequenly, he EKF algorihm esimaes he belief as follows [38]: µ = g(u,µ 1 ) Σ = G Σ 1 G T +R K = Σ H T (H Σ H T +Q ) 1 µ = µ +K (z h(µ )) Σ = (I K H )Σ bel(x ) = N(x ;µ,σ ) (9) (11) where G and H sand for Jacobians and N is he Gaussian disribuion. The underlying moion model of he probabilisic sysem defines he sae updae equaion, and hence he ransiion beween adjacen saes. Alhough in roboic applicaions, here are a variey of moion models depending on he ype of he robo and he kinemaics of is moving pars, in our case of a single independenly moving objec, he underlying moion model is relaively easily defined. The objec moves wih respec o eiher consan posiion, consan velociy or consan acceleraion model. In vision lieraure, consan velociy moion model is usually uilized o model moion of free moving hand-held cameras and rigid objecs [40], [35], [41], [42]. In consan velociy moion model, he sae is composed of posiion and velociy of he objec. The velociy beween consecuive ime insans is he same up o an addiive noise erm. This noise erm accouns for any possible acceleraion caused by sysem dynamics or exernal influences. On he oher hand, he posiion is updaed by simple addiion of previous posiion, change in posiion (velociy imes dela ime) and a noise erm. The aim of he Bayes filer (acually he EKF) is o esimae he saes, i.e. R do, do and associaed velociies, by using he observaions [X odi,y odi,z odi ] T and [x ovi, ] T. In consan velociy moion model, he sae updae equaions, relaing consecuive saes, can be wrien as follows: ρ do θ do φ do xdo ydo zdo ρ do θ do φ do ṫ xdo ṫ ydo ṫ zdo = = ρ do θ do φ do xdo ydo zdo ρ do θ do φ do ṫ xdo ṫ ydo ṫ zdo ǫ ii ρ do θ do φ do ṫ xdo ṫ ydo ṫ zdo 1 +ǫ i (12) The firs line of (12) performs posiion updae by adding previous posiion and posiion updae, whereas he second line sands for he conservaion of velociy up o an addiive noise erm. I should be noed ha ime difference ( ) beween consecuive updaes is 1 frames. On he oher hand, measuremen equaions relaing curren saes and curren measuremens are exacly same as (1) and (2), excep an addiive noise erm o accoun for he measuremen noise. For N objec poins, 5N 1 measuremens are obained by concaenaing each sensor measuremens: α i X odi Y odi Z odi x ovi 1 = [R do ] X o i +[ do ] +ε i Z oi X oi = K v R vd [R do ] +[ do ] + vd (13) where [X oi,, ] T represens 3D coordinaes of i h objec poin wih respec o objec reference frame and subscrip denoes ime. As a final remark, insead of a 5N 1 measuremen vecor, a possible combinaion ha yields a 5 1 measuremen vecor could be considered. In [43], i is shown for he linear Kalman case ha he former approach is more flexible and compuaionally more efficien for ime-varying noise characerisics and increased number of measuremens. Alhough, he noise on measuremens are highly dependen on used vision and deph sensors, he scene characerisics and he feaure maching algorihm used, wihou loss of generaliy, +ε ii 2133

6 he measuremen noise covariance marix Q is designed in he form of a diagonal marix of size 5N 5N where each enry specifies variance of associaed measuremen: Q = diag(σ 2 XYZ,σ 2 pix) (14) A general discussion on he noise model for he Kinec sensor can be found in [44]. Finally, he measuremens can be associaed beween consecuive ime insans using he feaure racking algorihm in [4]. IV. TEST RESULTS The performance of he proposed RGBD daa based pose esimaion algorihm is compared o ha of mehods uilizing single sensor approaches. Please remember ha, as deailed in Secion I, he mehod using 3D measuremens esimaes objec pose using (1). To his aim quaernion based approach [45], which is widely used in he associaed lieraure, is adoped. The algorihm minimizes he 3D error ha is he difference beween ransformed objec coordinaes and 3D measuremens by he deph sensor [X odi,y odi,z odi ] T : e 3D = X odi Y odi Z odi [R do ] X o i +[ do ] (15) On he oher hand, he approach depending on 2D measuremens o esimae pose uses (2) and minimizes he reprojecion error beween pixel measuremens [x ovi, ] T and objec coordinaes [X oi,, ] T projeced on he image plane: e 2D = [ xovi ] [ xovi ],P (16) The approach finds iniial pose esimae using PnP algorihm [21] and refines i furher using LM opimizaion [34]. Noe ha hese single sensor algorihms are seleced based on he observaion ha many end-o-end 3D rackers uilize hese mehods as he core pose esimaion rouine. In order o analyze he performance of he mehods, an arificial es scenario is designed. To his aim, he Face daa se, wih he 3D model shown in Figure 4, is used. Since he model is composed of housands of poins, random 20 poins are seleced as [X oi,, ] T and uilized for racking. The iniial saes a ime 0 are seleced as follows: ρ do θ do φ do xdo ydo zdo ρ do θ do φ do ṫ xdo ṫ ydo ṫ zdo 0 0 = = rad 1.4 rad rad 50 mm 50 mm 2000 mm rad/frame rad/frame rad/frame mm/frame mm/frame mm/frame (17) Fig. 4: 3D model of he Face sequence. The saes a consecuive ime insans are obained according o (12). Similarly, (13) is used o generae he measuremens[x odi,y odi,z odi ] T and[x ovi, ] T a each ime insan for he poins seleced from he model. The measuremen noise variances σxyz 2 and σ2 pix are 10 mm and 0.5 pixels for 3D and 2D measuremens, respecively. The sequence consiss of 100 frames, herefore, we simulae a movemen of he head from lef o righ wih a dominan moion in he y-axis. The resuls are obained by Mone Carlo simulaions composed of 50 rials. The sensors are calibraed inernally and exernally by using he procedure deailed in [9]. Moreover, he iniial sae is assumed o be known for all mehods. Proposed mehod and single sensor approaches are compared in erms of 2D reprojecion error (16), 3D error (15) and deviaion from groundruh pose parameers available. Mean error values are obained by averaging associaed values for all racked objec poins in a frame. 2D reprojecion errors for mehods are shown in Table I. TABLE I: Mean reprojecion errors (in pixels). Mehod Fusion 2D only 3D only Error By consrucion, he algorihm uilizing vision sensor minimizes he reprojecion error. Therefore, in erms of reprojecion error i gives he bes performance. On he oher hand, 3D errors are obained as shown in Table II. TABLE II: Mean 3D errors (in mm). Mehod Fusion 2D only 3D only Error Similarly, mehod uilizing mere deph sensor measuremens minimizes 3D error; and hence i gives bes performance in erms of his meric. Finally, mean pose esimaion error values are abulaed in Table III. The sensor fusion approach minimizes 2D and 3D errors simulaneously and performs more accurae pose esimaion. Therefore, he proposed fusionbased mehod is much more reliable compared o single sensor based algorihms. 2134

7 TABLE III: Mean racking errors. Mehod Fusion 2D only 3D only roaion-x (mrad) roaion-y (mrad) roaion-z (mrad) ranslaion-x (mm) ranslaion-y (mm) ranslaion-z (mm) Insead of PnP and quaernion based algorihms, EKF is also uilized o perform pose esimaion using single sensor inpus. The sae updae equaion is exacly same as (12), however 2N 1 and 3N 1 measuremen vecors for mehods uilizing 2D-3D and 3D-3D correspondences are obained using he firs and second lines of (13) respecively. The pose esimaion accuracies are as follows: TABLE IV: Mean racking errors for single sensor mehods. Mehod 2D only EKF 3D only EKF roaion-x (mrad) roaion-y (mrad) roaion-z (mrad) ranslaion-x (mm) ranslaion-y (mm) ranslaion-z (mm) I is clear from Table IV ha, alhough he mehods are much more accurae han heir insananeous counerpars depiced in Table III, hey canno perform beer ha he proposed sensor fusion approach. Figure 5 illusraes he pose esimaion errors and associaed variance esimaes by he filer (please noe ha only values for dominan y-direcional roaion esimae is given for space resricions). As he filer is updaed error values converges owards zero, also he variance of he esimae is decreases, as expeced. Therefore, we can conclude ha variances can be uilized as a figure of meri for deducion of qualiies of esimaes. Finally, i is observed ha he filer can olerae up o 5% error during iniializaion and canno converge afer ha poin. V. CONCLUSION Deph and vision sensors provide daa having compleely differen saisics and an opimal racking mehod should handle his issue by considering he underlying noise models. Moreover, in 3D objec pose esimaion, he pose esimaes need o be emporally consisen in order o reduce jier and generae visually more pleasan racks. Considering hese precondiions, EKF comes as a powerful soluion ha can fuse RGBD daa in order o solve nonlinear pose esimaion problem. Hence, in his paper, saring from he basic sae updae and measuremen equaions, he proposed sensor fusion approach ha adops consan velociy moion model is deailed. The performance of he proposed formulaion is analyzed in erms accuracy compared o single sensor approaches. I is observed ha alhough single sensor approaches successfully minimize errors in 2D and 3D spaces, heir accuracies are much lower han he sensor fusion approach. ACKNOWLEDGEMENT This work is parially suppored by The Scienific and Technological Research Council of Turkey. O. Serdar Gedik was a Middle Eas Technical Universiy Deparmen of Elecrical and Elecronics Engineering a he ime of his work. REFERENCES [1] NASA, Nasa global hawks aid uav-o-uav refueling projec, hp:// accessed: 07/04/2014. [2] BMW, Bmw augmened realiy, hp:// accessed: 07/04/2014. [3] MaerPor, Maerpor 3d models for real inerior spaces, hp://maerpor.com/, accessed: 07/04/2014. [4] O. S. Gedik and A. A. Alaan, 3d rigid body racking using vision and deph sensors, IEEE Transacions on Cyberneics Par B, vol. 43, no. 5, pp , (a) Roaion-y Error (b) Roaion-y Variance Fig. 5: Error and variance values. 2135

8 [5] MESA, Mesa imaging, hp:// accessed: 07/04/2014. [6] Microsof, Microsof kinec, hp:// accessed: 04/04/2013. [7] ASUS, Mulimedia xion pro, hp:// accessed: 07/04/2014. [8] J. Surm, S. Magnena, N. Engelhard, F. Pomerleau, F. Colas, W. Burgard, D. Cremers, and R. Siegwar, Towards a benchmark for rgb-d slam evaluaion, in RGB-D Workshop on Advanced Reasoning wih Deph Cameras, [9] O. S. Gedik and A. A. Alaan, Fusing 2d and 3d clues for 3d racking using visual and range daa, in The Inernaional Conference on Informaion Fusion, [10] R. Harley and A. Zisserman, Muliple View Geomery in Compuer Vision. Cambridge Universiy Press, [11] J. Lima, F. Simes, L. Figueiredo, V. Teichrieb, and J. Kelner, Online monocular markerless 3d racking for augmened realiy, in Abordagens Pricas de Realidade Virual e Aumenada: SVR, [12] D. F. DeMenhon and L. S. Davis, Model-based objec pose in 25 lines of code, Inernaional Journal of Compuer Vision, vol. 15, no. 1 2, pp , [13] P. David, D. F. DeMenhon, R. Duraiswami, and H. Same, Sofposi: Simulaneous pose and correspondence deerminaion, in European Conference on Compuer Vision. Springer, [14] S. Gold and S. Rangarajan, A graduaed assignmen algorihm for graph maching, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 18, no. 4, pp , [15] R. I. Harley, Minimizing algebraic error in geomeric esimaion problems, in IEEE Inernaional Conference on Compuer Vision, [16] C. Lu and G. Hager, Fas and globally convergen pose esimaion from video images, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 22, no. 6, pp , [17] P. D. Fiore, Efficien linear soluion of exerior orienaion, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 23, no. 2, pp , [18] A. J. Gruner, Das pohenoische problem in erweierer gesal nebs bber seine anwendungen in der geodasie, in Gruners Archiv fur Maheinaik und Physik Band 1, [19] R. M. Haralick, C. Lee, and M. Nolle, Analysis and soluions of he hree poin perspecive pose esimaion problem, in IEEE Conference on Compuer Vision and Paern Recogniion, [20] L. Quan and Z. Lan, Linear n-poin camera pose deerminaion, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 21, no. 8, pp , [21] S. Ohayon and E. Rivlin, Robus 3d head racking using camera pose esimaion, in IEEE Inernaional Conference on Paern Recogniion, [22] F. Moreno-Noguer, L. V., and P. Fua, Accurae non-ieraive o(n) soluion o he pnp problem, in IEEE Inernaional Conference on Compuer Vision, [23] E. Murphy-Chuorian and M. M. Trivedi, Head pose esimaion in compuer vision: A survey, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 31, no. 4, pp , [24] C. Shan and W. Chen, Head pose esimaion using specral regression discriminan analysis, in IEEE Conference on Compuer Vision and Paern Recogniion, [25] C. Canon-Ferrer, C. J. R., and M. Pardas, Head orienaion esimaion using paricle filering in muliview scenarios, in Mulimodal Technologies for Percepion of Humans. Springer, [26] M. Voi, K. Nickel, and R. Siefelhagen, Neural nework-based head pose esimaion and muli-view fusion, in Mulimodal Technologies for Percepion of Humans. Springer, [27] D. A. Simon, M. Heber, and T. Kanade, Real-ime 3-d pose esimaion using a high-speed range sensor, in IEEE Conference on Roboics and Auomaion, [28] P. J. Besl and N. D. McKay, A mehod for regisraion of 3d shapes, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 14, no , [29] L. Shang, B. Jasiobedzki, and M. Greenspan, Model-based racking by classificaion in a iny discree pose space, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 29, no. 6, pp , [30] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shoon, S. Hodges, D. Freeman, A. Davison, and A. Fizgibbon, Kinecfusion: Real-ime 3d reconsrucion and ineracion using a moving deph camera, in ACM Symposium on User Inerface Sofware and Technology, 201. [31] N. Gelfand, N. J. Mira, L. J. Guibas, and H. Pomann, Robus global regisraion, in Eurographics Symposium on Geomery Processing, [32] B. Sreckel, B. Barczak, R. Koch, and A. Kolb, Supporing srucure from moion wih a 3d-range-camera, Lecure Noes in Compuer Science, vol. 4522, pp , [33] M. A. Fischler and B. C. Bolles, Random sample consensus: A paradigm for model fiing wih applicaions o image analysis and auomaed carography, Communicaions of he ACM, vol. 24, no. 6, pp , [34] W. H. Press, S. A. Teukolsky, W. T. Veerling, and B. P. Flannery, Numerical Recipes in C - The Ar of Scienific Compuing. Cambridge Universiy Press, [35] M. Krainin, P. Henry, X. Ren, and D. Fox, Manipulaor and objec racking for in-hand 3d objec modeling, The Inernaional Journal of Roboics Research, vol. 30, no. 11, pp , [36] F. Seinbrcker, J. Surm, and D. Cremers, Real-ime visual odomery from dense rgb-d images, in IEEE Inernaional Conference on Compuer Vision Workshops, [37] J. Sckler and S. Behnke, Model learning and real-ime racking using muli-resoluion surfel maps, in Tweny Sixh AAAI Conference on Arificial Inelligence, [38] S. Thrun, W. Burgard, and D. Fox, Probabilisic Roboics. MIT Press, [39] R. E. Kalman, A new approach o linear filering and predicion problems, Transacions of he ASME Journal of Basic Engineering, vol. 82, no. Series D, pp , [40] A. Davison, I. D. Reid, M. N. D, and O. Sasse, Monoslam: Realime single camera slam, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 29, no. 6, pp , [41] K. Nickels and S. Huchinson, Weighing observaions: The use of kinemaic models in objec racking, in IEEE Inernaional Conference on Roboics and Auomaion, [42] G. Taylor and L. Kleeman, Fusion of mulimodal visual cues for modelbased objec racking, in Ausralasian Conference on Roboics and Auomaion, [43] Q. Gan and C. J. Harris, Comparison of wo measuremen fusion mehods for kalman-filer-based mulisensor daa fusion, IEEE Transacions on Aerospace and Elecronic Sysems, vol. 37, no. 1, pp , [44] C. V. Nguyen, S. Izadi, and D. Lovell, Modeling kinec sensor noise for improved 3d reconsrucion and racking, in Inernaional Conference on 3D Imaging, Modeling, Processing, Visualizaion and Transmission, [45] R. Jain, R. Kasuri, and B. G. Schunck, Machine Vision. McGraw-Hill,