Geometric Primitive Refinement for Structured Light Cameras

Transcription

1 Self Archve Verson Cte ths artcle as: Fuersattel, P., Placht, S., Maer, A. Ress, C - Geometrc Prmtve Refnement for Structured Lght Cameras. Machne Vson and Applcatons 2018) 29: 313. Geometrc Prmtve Refnement for Structured Lght Cameras Peter Fuersattel Smon Placht Andreas Maer Chrstan Ress Receved: date / Accepted: date Abstract 3-D camera systems are useful sensors for several hgher level vson tasks lke navgaton, envronment mappng or dmensonng. However, the raw 3-D data s for many algorthms not the best representaton. Instead, many methods rely on a more abstract scene descrpton, where the scene s represented as a collecton of geometrc prmtves lke planes, spheres or even more complex models. These prmtves are commonly estmated on ndvdual pont measurements whch are drectly affected by the measurement errors of the sensor. Ths paper proposes a method for refnng the parameters of geometrc prmtves for structured lght cameras wth spatally varyng patterns. In contrast to fttng the model to a set of 3-D pont measurements, we propose to use all nformaton that belongs to a partcular object smultaneously to drectly ft the model to the mage, wthout the detour of calculatng dspartes. To ths end, we propose a novel calbraton procedure whch recovers the unknown nternal parameters of the range sensors and reconstructs the unknown projected pattern. Ths s partcularly necessary for consumer structured lght sensors whose nternals are not aval- Ths work was supported n part by the Research Tranng Group 1773 Heterogeneous Image Systems, funded by the German Research Foundaton DFG), and n part by the Erlangen Graduate School n Advanced Optcal Technologes SAOT) by the German Research Foundaton DFG) n the framework of the excellence ntatve. Peter Fuersattel 1, Andreas Maer 1, Chrstan Ress 1 {frst}.{last}@fau.de Smon Placht 2 smon.placht@metrlus.de 1 Fredrch-Alexander Unversty Erlangen-Nuremberg 2 Metrlus GmbH, Erlangen able to the user. After calbraton, a coarse model ft s consderably refned by comparng the observed structured lght dot pattern wth a predcted vrtual vew of the projected vrtual pattern. The calbraton and the refnement methods are evaluated on three geometrc prmtves: planes, spheres and cubods. The orentatons of the plane normals are mproved by more than 60 %, and plane dstances by more than 30 % compared to the baselne. Furthermore, the ntal parameters of spheres and cubods are refned by more than 50 % and 30 %. The method also operates robustly on hghly textured plane segments, and at ranges that have not been consdered durng calbraton. Keywords Structured lght Range magng Geometrc Prmtves 1 Introducton Accurate scene nformaton s a crtcal component for numerous vson-based applcatons, such as dmensonng, robot navgaton, floor detecton, SLAM algorthms or generally any map buldng task. These applcatons greatly beneft from the recent prolferaton of low-cost cameras that capture 2-D mages wth addtonal depth nformaton [20, 4, 22]. Well-known depth cameras are based on Tme-of-Flght, stereo vson, and structured lght. These cameras capture dense, ordered pont clouds of the scene at hgh frame rates wth reasonable accuracy. Hgher level vson tasks lke robot navgaton or mappng oftentmes compute an ntermedate representaton of the data. The segmentaton and accurate detecton of planes s a common task. For example, n the context of robotcs, several algorthms for synchronous localzaton and mappng SLAM) operate on 3-D planes

2 2 Peter Fuersattel et al. that have been detected and segmented from the raw sensory nformaton [22, 23, 4, 3]. Plane representatons have also bee used for the calbraton of range sensors to color sensors [11, 9]. The extrapolaton of mssng measurements at object boundares as demonstrated by Moerwald et al. s another example for the usefulness of abstract representatons of objects n the scene [17]. Fttng models to specfc regons n the captured pont cloud s also useful n a dmensonng context, e.g. f the sze of box-lke objects needs to be measured. Naturally ths apples to all types of models whch can be ft to a lst of 3-D measurements, for example n a least-squares fashon or va RANSAC. Abstract representatons are popular because they address three challenges of workng wth 3-D data: 1. Hgher level vson algorthms oftentmes do not operate on the raw sensory measurements, but requre a model of the envronment anyways. Furthermore, abstract representatons smplfy many common calculatons lke determnng the sze of objects or computng the ntersectons of multple objects. 2. The addtonal depth dmenson ncreases the amount of data to be processed whch may be a bottleneck, especally n real-tme applcatons lke SLAM). By computng a smplfed geometrc representaton, the data rate can be consderably reduced. 3. Current range cameras provde only lmted accuracy. For structured lght and Tme-of-Flght cameras, these errors range from several mllmeters up to multple centmeters [14, 8]. Fttng models of geometrc prmtves to segments of the pont cloud can reduce the mpact of nose on subsequent calculatons, especally f the measurement errors stem from temporal nose sources. In ths work, we present a hghly accurate method for fttng a 3-D model to a segment n a pont cloud usng a consumer structured lght camera. Well-known examples of ths class of sensors are the Mcrosoft Knect, Asus Xton and the more recently released Orbbec Astra 1. These cameras use a sngle nfrared camera to capture a projector-generated nfrared dot pattern. The dstorton of the dot pattern s used to compute depth. One advantage of structured lght sensors over stereo systems s that they provde dense depth data on surfaces wth homogeneous texture. Typcally, structured lght cameras calculate dstances va block matchng of small patches. In contrast to that, we propose a method that explots the complete 2-D mage nformaton that represents an object to fnd an accurate parametrc object for ths object. 1 Our method s based on the dea that, for any parametrc model n 3-D, one can smulate the projecton and observaton of the emtted dot pattern. We propose to create a vrtual vew of the object based on a model that defnes the object. If the model descrbes the observed object perfectly, then the vrtual mage and the observed mage wll be dentcal. Thus, by measurng the smlarty between these two mages and adjustng the model parameters such that the smlarty ncreases, one can optmze for more accurate parameters. A fundamental requrement for creatng such vrtual vews s the knowledge of all nternal parameters that descrbe the structured lght sensor: a model of the projector, the spatal relaton between projector and camera and the emtted dot pattern. The challenge wth off-the-shelf structured lght sensors, lke those mentoned above, s that the user typcally has no access to these parameters. To tackle ths problem, we propose a novel calbraton method, whch s able to recover the ntrnsc parameters of the projector, the spatal relaton between the projector and the camera, and to reconstruct the unknown emtted dot pattern from two or more mages. The contrbutons of ths work consst of three parts: 1. We present a novel calbraton technque to estmate pnhole model parameters for the projector, the extrnsc parameters of camera and projector, and the unknown dot pattern. 2. We present a new refnement approach for geometrc prmtves for structured lght cameras. In contrast to the regular pxel-wse dsparty calculaton, we propose to use all pxels that belong to the object smultaneously. Ths mage nformaton s combned wth constrants posed by the model s shape to refne the ntal parameters. 3. We show that ntal estmates for three dfferent geometrc prmtves, namely planes, spheres and cubods, can be refned sgnfcantly wth the proposed method. In the case of planes, we show that the angular and dstance accuracy of plane models are ncreased by more than 60 % and 30 % compared to the baselne. Intal estmates of spheres can be mproved by up to 50 %. Furthermore, even complex models, lke cubods can be refned by up to 30 %. In Secton 2, we present related work. The notaton used n ths work s explaned n Secton 3. Secton 4 contans a detaled descrpton on the calbraton method. In Secton 5 we gve nformaton on how to obtan an ntalzaton for an object whch can be refned wth the method descrbed n Secton 6. The performance of the method s evaluated n Secton 7. We summarze and conclude our fndngs n Secton 8.

3 Geometrc Prmtve Refnement for Structured Lght Cameras 3 Intal parametrc model Coarse model fttng Depth Compare θ Render vrtual vew Π c, Π p, R cp, S Sensor Model IR Camera Adjust model parameters by θ θ Refned parametrc model Fg. 1: Geometrc Prmtve Refnement. Parameters θ of an ntal parametrc model are refned n an teratve process. By creatng vrtual vews of the object and comparng t to the observed mage, θ s adjusted such that the smlarty ncreases and an optmal parameter set θ can be obtaned. The sensor model, that s summarzed wth the parameters Π c, Π p, R cp, S, s obtaned wth the calbraton method descrbed n Secton 4. 2 Related Work A structured lght camera system s defned by a number of parameters: the ntrnsc parameters of the camera and the projector, ther spatal relaton and fnally the projected pattern. In custom setups, these parameters can ether be drectly controlled by the user, or obtaned va calbraton. Moreno and Taubn publshed a calbraton toolbox for such systems to fnd the spatal relaton by usng a calbraton target [16]. Ye and Song recently proposed a method for refnng an ntal calbraton result based on control ponts n 3-D space [27]. Yamazak et al. presented a method for estmatng the spatal relaton between the camera and the projector as well as the ntrnsc parameters of both devces [26]. The method does not requre a calbraton target but s consequently not able to determne the scale. Another target-less approach for setups wth at least two cameras and a projector have been proposed by Brd and Papankolopoulos [2]. The above-mentoned methods are desgned for custom structured lght systems, or setups whch allow the user to control the projected patterns. However, these methods are not applcable for off-the-shelf sensors, lke the Mcrosoft Knect or Orbbec Astra, as these cameras do not provde access to ther nternal parameters. In ths work we relax these requrements and present a method whch s capable of calbratng a generc structured lght camera wth an unknown pattern. In prncple, ths work s closely related to the prmtve detecton and segmentaton n pont clouds. However, a sgnfcant dfference les n the focus. In our work, the objectve s to obtan an estmate for the locaton and the parameters that defne an object as accurate as possble. In contrast to that, the prmary objectve of the detecton methods les n a fast, coarse segmentaton, typcally wth a strong focus on real-tme applcatons. There exsts a large body of work on geometrc prmtve detecton for pont clouds that have been captured wth, for example, a structured lght camera. A revew on pont cloud segmentaton can be found n [18]. Poppnga et al. segment a scene nto planes by performng regon growng on the 3-D pont cloud [20]. The authors present an effcent method to calculate least squares fts of planes to the ponts. The authors of [12] extend ths method by a multlateral flterng step to ncrease robustness aganst measurement nose. The authors also propose an alternatve, less accurate segmentaton method to reduce computatonal complexty. Trevor et al. reduce the number of plane fts by segmentng the mage frst and only fttng planes to regons whch fulfll a sze crteron [24]. Mörwald et al. dentfy regons that can be modeled wth B-Splnes [17]. Ths method jontly uses range and color nformaton to calculate mssng range nformaton n the scene, whch may happen n pxels where the correspondence search fals. Georgev et al. propose to frst search for lne segments n the pont cloud to reduce the search space [10]. In a second step, these lne segments are analyzed and combned to obtan the plane segments n the scene. In

4 4 Peter Fuersattel et al. [6] a real-tme capable, graph-based approach s presented. Graph nodes represent non-overlappng mage regons. These nodes are analyzed separately for planarty and merged f they belong to the same segment. Then, regon growng s used to determne the exact pxels whch belong to the ndvdual models. Borrmann et al. present a new Hough transform accumulator for 3-D plane detecton and compare t to other varants of the Hough transform [5]. The plane estmate n the methods above s relatvely straghtforward. It oftentmes conssts of a least-squares ft to the 3-D pont cloud usng sngular value decomposton or varants thereof. However, such a fast approach s qute prone to outlers or other estmaton errors. All n all, applcatons can beneft from our method f hgh accuracy s requred, but runtme requrements are somewhat less strngent. One mportant dfference to the methods above les n the fact that our approach does not operate on the 3-D ponts, but nstead drectly on the dot pattern of the structured lght sensor. Ths allows to jontly ft several dots of the pattern to a geometrc prmtve, and hence to avod ndvdual errors n the stereo matchng process. The dea to drectly operate on the dot pattern has also been used n other applcatons. Fanello et al. pose the correspondence search problem as a classfcaton problem, and solve t for each pxel ndvdually wth random forests [21]. The projector pattern s obtaned from a calbraton and tranng phase. One notable dfference of ths work to ours s that we do not operate on each dot ndvdually. Instead, we nclude non-local cues by jontly optmzng for all dots on one surface. McIlroy et al. also operate drectly on the dot pattern [15]. The authors propose a low-cost tracker based on the Knect camera. The authors demonstrate how the projecton of the dot pattern onto a fxed mult-planar surface can be used to determne the pose of the projector. We use a smlar dea, but we solve for the projecton surface usng a fxed camera-projector setup. Abstract representatons of geometrc prmtves, e.g. planes, have been used by a number of works on hgherlevel vson tasks. Methods of ths type can potentally beneft from the hgh accuracy of the presented algorthm. An extenson of the work by Holz et al. [12] uses a map of planes for localzaton or place recognton [7]. In [25], planes are used as features to create a map of the envronment. These planes are used to effectvely reduce the computatonal complexty of the mappng process and to ncrease the accuracy of the map and localzaton tasks. Brk et al. demonstrate how planar surface patches can be used to estmate the robot moton effcently va a determnstc, non-teratve method [3]. Bswas et al. present a RANSAC-based plane segmentaton for obstacle avodance and navgaton, whch reles only on depth nformaton [4]. Taguch et al. show that the use of planes s mportant to acheve hgh frame rates for 3D reconstructon wth the Knect sensor [23]. Salas et al. present a SLAM system that recovers the boundares of planes n the scene over tme [22]. 3 Notaton We denote mages by bold captal letters, for example mage I. Bold lowercase letters denote vectors, for example x = X, Y, Z). If requred, a superscrpt specfes the coordnate system of the vector, for example, x c) denotes a pont as seen from camera c. Geometrc prmtves, for example planes, spheres or cylnders, are represented as parametrc 3-D models. All parameters that descrbe such a model are summarzed n a parameter vector θ. In the case of a plane, we use the Hesse normal form to descrbe a plane wth four parameters: three values to represent the unt normal, and one value to represent the dstance between the orgn and the plane. A translaton vector and a rotaton matrx form the rgd body transformaton R cp that, for example, relates the coordnate systems of the camera and the projector. We use Π to descrbe a projecton from 3-D space to the 2-D mage doman. Ths projecton s mplemented as a pnhole camera model, or a pnhole camera model combned wth a lens dstorton model. Π 1 θ descrbes an nverse projecton. In ths work, nverse projectons are mplemented as the ntersecton of a ray wth a parametrc 3-D model defned by θ. 4 Calbraton of the Structured Lght Camera and Reconstructon of the Dot Pattern The complete calbraton process s llustrated n Fgure 2. We frst descrbe the acquston of a set of calbraton mages n Secton 4.1. In Secton 4.2, we demonstrate how a coarse ntalzaton of the sensor s extrnsc parameters can be approxmated. Next, we create an ntal reconstructon of the dot pattern as descrbed n Secton 4.3. Hereafter, we show how the ntal parameters and the ntal dot pattern can be refned n a non-lnear optmzaton scheme based on multple calbraton mages. The refnement s descrbed n Secton 4.5.

5 Geometrc Prmtve Refnement for Structured Lght Cameras 5 Secton 4.2 Secton 4.3 Secton 4.4 Secton 4.5. contnue Coarse estmate of R cp and f p Calculate ntal dot pattern S Render vrtual mages Observed Estmate gradents mages Update R cp and f p, update S Evaluate cost functon Camera ntrnscs, calbraton mages converged Refned R cp, f p and S Fg. 2: Calbraton process. Startng from a set of calbraton mages and camera ntrnscs, the extrnsc parameters, the ntrnsc parameters of the projector and the unknown dot pattern are estmated. 4.1 Calbraton Data and Reference Dstance Informaton x 2 x 1 P In ths secton we descrbe the calbraton data and how t can be acqured. The calbraton procedure requres at least two mages, each consstng of a dot pattern mage and a correspondng pont cloud. The mages need to show multple planar scenes. The planes may be parallel but not coplanar. A homogeneous whte wall captured at dfferent dstances wll ensure the best possble estmaton of the nternal parameters and of the dot pattern. Generally, more mages captured at dfferent dstances wll result n better estmates. The calbraton protocol requres that the poston and orentaton of an maged plane s known. Theoretcally, one can use the dstance nformaton of the pont cloud to estmate the plane model. However more accurate plane models can be obtaned wth calbraton patterns. The dot pattern s captured best n a two-step process that conssts of estmatng the plane wth a calbraton pattern, and removng the calbraton pattern before capturng the dot pattern. In prelmnary experments, we have compared dfferent calbraton patterns wth each other, ncludng checkerboards wth ROCHADE [19] and RUNETags [1] wth the mplementaton provded by the authors. In these experments, checkerboards were detected over a wder range of dstances and more accurately than RUNETags, whch s why we use checkerboards. 4.2 Coarse Intalzaton of the Intrnsc and Extrnsc Parameters P x 2 r 2 r 1 A frst coarse calbraton s obtaned n two steps. Frst, we estmate the relatve poston between projector and camera R pc. In the second step, we calculate the parax 1 b Fg. 3: Intal baselne estmaton. The rays r generated from two correspondng pont pars x and x are used to calculate the ntal baselne b. meters of a pnhole camera model that represents the projector. We slghtly smplfy the estmaton of the relatve poston between camera and projector. Snce the prncpal rays of camera and projector are parallel n almost all offthe-shelf structured lght cameras, the spatal relaton R pc of camera and projector reduces to a translaton b. We call b the baselne. The estmaton of the baselne s vsualzed n Fgure 3. Here, the projector emts rays r, that eventually ht a plane. The llustraton shows that the same ray r can also be calculated from two correspondng ponts x and x whch le on two dfferent planes P and P. Thus, the center of projecton of the projector P, and consequently the baselne b, can be computed by fndng the ntersecton of two or more rays. For ths step the ntrnsc parameters of the camera Π c are assumed to be known or that they can be calculated n a separate calbraton step [28]. The dervaton requres that the postons of the two planes P and P are known. Ther parameters can be C P

6 6 Peter Fuersattel et al. X p) x p) = x c) + b x p) = R pc x c) ) W 2 P Projector mage f P p p) b C p c) Z p) Observed mage Fg. 4: Focal length ntalzaton. The projector s focal length f p s computed from the known camera ntrnscs, a known plane and the ntal baselne b. Shaded areas vsualze the two felds of vew for camera C and projector P. S P p p) r p) p c) Fg. 5: Vrtual mage generaton. The ray r p) for the pont p p) s ntersected wth the plane P to obtan x p). Ths pont s transformed to the coordnate system of the camera, and fnally projected wth Π c to get p c). C I estmated from the structured lght depth measurements or preferred) from a calbraton pattern attached to ther surfaces. Pont correspondences x, x ) are obtaned va block matchng. If camera and projector postons dffer only n the x-coordnate, the block search can be constraned to the x-axs. Ths approach can be extended easly to more than two rays to ncrease the accuracy of the estmate. In practce, fve to ten rays were suffcent. Best ntalzatons are obtaned f the correspondences are well spread n the mage, for example equally among the four quadrants of the mage. The approxmaton of the baselne allows the estmaton of the pnhole model parameters of the projector. We assume that the projector has an deal lens and that the prncpal pont s at the mage center. Hence, fndng the projector model reduces to estmatng the focal length. We calculate the focal length such that a pxel that s observed n a corner of the camera mage s also located n the same corner of the projector s mage. Fgure 4 llustrates ths scenaro. In order to capture the complete observed pattern, we choose the corner p c) that corresponds to the projecton of a 3-D pont x c) = X c), Y c), Z c)) that maxmzes the dstance to the projector. Wth x p) = x c) + b and ts respectve projecton p p), the focal length can be computed drectly wth the pnhole camera model: f p = W W ) Z p). 1) 2 X p) Computng f p requres the wdth W of the fxed mage to be known. In practce, W should be set to a value equal or larger than the wdth of the camera mage. The resultng focal length drectly depends on the dstance between the observed plane and the camera, wth closer planes resultng n a smaller focal length. Therefore, we use the calbraton mage wth the smallest plane-camera dstance for calculatng f p n order to nclude the largest possble secton of the projected dot pattern. 4.3 Dot Pattern Reconstructon In ths secton, we descrbe how the statc mage, that the emtter projects nto the scene, can be reconstructed. We call the reconstructed mage the vrtual dot pattern S. Below we descrbe how the ntenstes for all pxels n S can be obtaned from correspondng pxels n I by usng the prevously calculated ntrnsc and extrnsc parameters. The reconstructon process s vsualzed n Fgure 5. Here, the ntensty for a pont p p) correspondng pont p c) s calculated from ts n I. The coordnates p c) are obtaned by ntersectng r wth the plane, transformng the ntersecton to the camera s coordnate system, and fnally by projectng t onto the mage plane. The ntensty n the vrtual dot pattern at p p) s now nterpolated from I. Ths s expressed by nterp ). Mathematcally, the ntensty lookup for a sngle pxel of S s gven by ) ) S p p), θ = nterp I, p c) )))) = nterp I, Π c R pc.2) Π 1 p,θ p p) The estmaton of the ntal dot pattern can be extended to use multple mages, analogously as descrbed n Secton 4.2. However, smlar as n Secton 4.2, t

7 Geometrc Prmtve Refnement for Structured Lght Cameras 7 turned out that a coarse estmate of the pattern s suffcent f an addtonal refnement step s performed. 4.4 Vrtual Image Generaton Wth known dot pattern, extrnsc and ntrnsc parameters, t s possble to generate a vrtual mage V of a parametrc model. In the context of the calbraton, the goal s to generate a vrtual mage of a plane n the calbraton scene. To ths end, we use the nverse reconstructon ppelne gven n Equaton 2, wth R cp beng the nverse of R pc. The plane s defned by θ. By usng the prevously calculated vrtual dot pattern S as source for the lookup, pxel values for all coordnates p c) V can be obtaned: p p) ), θ = nterp S, Π p R cp Π 1 c,θ p c) )))). 3) Note that the mage generaton ppelne expressed n Equaton 3 allows the adjustment of all parameters whch descrbe ether the scene or the structured lght sensor. Ths property wll be exploted for refnng the sensor parameters and the ntal vrtual dot pattern n Secton 4.5. Also note that although the calbraton operates on planes, the applcaton of the refnement can operate on arbtrary geometrc structures that can be formulated as a parametrc model. The sole requrement s the mplementaton of a method whch calculates the ntersecton of a ray wth the respectve model. The vrtual mage generaton can be parallelzed as all pxel values can be computed ndependently. Dependng on the used geometrc prmtve and camera propertes addtonal smplfcatons may become possble. For example, f planar regons are refned and lens dstorton s neglgble, then the mappng between dot pattern and vrtual mage can be represented by a homography. 4.5 Optmzaton of Intrnsc Parameters and the Dot Pattern The coarse calbraton s based only on a small number of pont correspondences. Durng refnement, the complete mage nformaton of M calbraton mages s utlzed to obtan more accurate ntrnsc and extrnsc parameters as well as a more accurate reconstructon of the dot pattern. We assume, that f the true sensor parameters are known, then t s possble to create vrtual vew of the scene that s dentcal to the observed camera mage. We use gradent descent to refne the rgd body transformaton R cp, focal length f p of the projector and the dot pattern S as seen from the projector. In each teraton, two steps are performed: frst, the refnement of the extrnsc parameters and the focal length, and second, an update of the dot pattern. Ths teraton s vsualzed n Fgure 2. Gradent descent requres a cost functon to measure the dssmlarty of two mages. To ths end, we calculate the mean squared dfferences between an observed mage I and a vrtual mage V, whch tself depends on a partcular f p and R cp. To ncrease the comparablty, each pxel s normalzed by the mean value of ts neghborhood. Ths can be mplemented effcently usng ntegral mages. The cost functon for a normalzed mage I and ts correspondng normalzed vrtual mage V s gven by e Rcp,f p I, V θ)) = 1 N N I V θ) ) 2. 4) In ths equaton, N denotes the number of pxels. We obtan the gradents numercally by alterng the current parameters ndvdually by small deltas. Note that the pxel-wse dfferences and the requred normalzaton prevent the usage of analytc dervatves. Once the gradents are known, the current parameter set s adjusted and used to estmate a new dot pattern. From each of the M calbraton mages, a dot pattern S j s derved as outlned n Secton 4.3. The updated dot pattern S s computed as the pxel-wse medan mage,.e., S = medan [S 1, S 2,..., S M ]). 5) The medan flter lmts the mpact of sensor nose on the dot pattern reconstructon. Durng the next teraton, S wll serve as the current dot pattern S. The optmal parameter set, consstng of R cp, f p and S can be obtaned by mnmzng Equaton 6. The functon mnmzes the dssmlarty between the M normalzed vrtual vews V j of known planes wth ther correspondng normalzed observatons I j. R cp, f p, S ) = argmn R cp,f p,s M j e Rcp,f p I j, V jθ j ) ) 6) Wth the aforementoned method, the projector parameters, baselne, rotaton and dot patterns can be estmated and saved for future use. Recalbraton s only requred f the camera/projector setup changes whch typcally does not occur durng operaton.

8 8 Peter Fuersattel et al. 5 Geometrc Prmtve Detecton The refnement method requres an ntal estmate of the object s model whose parameters should be refned. There are varous approaches for obtanng such an ntalzaton, for example a complete decomposton of the pont cloud nto geometrc prmtves, or a semautomatc segmentaton whch s based on seed values or a regon of nterest. In Secton 2, we lsted example works for fndng dfferent prmtves lke planes [20, 12, 24] or B-splne models [17]. In ths work we do not put any constrants on how the ntal parameters for a partcular model are obtaned. Instead, we only requre a parameter vector θ that descrbes the model analytcally. In ths work, three dfferent models are nvestgated: planes, cubods and spheres. A general lmtaton of the proposed method s that t assumes that the geometrc model s a suffcently accurate representaton of a pre-segmented area of the scene. Stuatons where, for example, parts of the prmtve are occluded by another object, have to be addressed n a separate processng step. In our mplementaton, we used masks to gnore such outler regons. 6 Geometrc Prmtve Refnement The core dea of the refnement algorthm s to create a vrtual vew of the object. If all parameters that descrbe the object are ether known or correctly estmated, the vrtual vew matches the observed mage. Hence, the objectve s to estmate the unknown parameters n a way that mnmzes the devatons between the vrtual vew and the observed mage. For generatng a vrtual vew, a complete model of the magng system s requred. In ths secton, we assume that the ntrnsc parameters of the camera and the projector, ther spatal relaton and the projected dot pattern have already been calculated wth the calbraton procedure descrbed n Secton 4. The parameters of an object n the scene, for example a plane, are summarzed n the parameter vector θ. In ths context, we assume that there exsts an ntal estmate of the object whch s obtaned by some ntal detecton algorthm. In ths secton, we present how these parameters can be further mproved, such that ther vrtual vew matches the observed mage more accurately. The optmal model parameters θ are obtaned va nonlnear optmzaton. Smlar as durng calbraton, we use a gradent descent approach to fnd the optmum. A metrc e s used to measure the dssmlarty between the observed mage I of a camera and a vrtual mage V θ). Here, we reuse a slghtly altered verson of the error metrc defned n Equaton 4. In the optmzaton phase of the calbraton procedure, ths error metrc s used to fnd R cp and fp based on fxed plane model parameters. Durng refnement of object parameters, we fx the camera s system parameters and vary the object parameters θ. Therefore, the dssmlarty n terms of the mean squared error for a gven θ s defned as: e R cp,f p I, V θ)) = 1 N N I V θ) ) 2. 7) The goal of the refnement step s to fnd parameters θ that maxmze the smlarty or mnmze the dssmlarty, respectvely) between I and V R cp, θ). More precsely, we seek θ = argmn θ er cp,f p I, V θ)) ). 8) Several effects, lke scene texture, varyng surface reflectance, or the angle of ncdence at lateral surfaces may negatvely affect the optmzaton n Equaton 8. Thus, to ncrease the robustness of the optmzaton, we frst normalze each pxel by dvdng t by the mean value of ts neghborhood. Ths can be mplemented effcently usng ntegral mages. Let I and V be the normalzed mages of I and V. Then, Equaton 8 becomes θ = argmn er cp θ,f p I, V θ)) ) = argmn θ 1 N ) N I V θ)) 2, 9) whch can be optmzed va gradent descent. We observed that a number of outlers can be expected at the border of a segment. Thus, applyng an addtonal eroson to the segment boundares ncreases the overall accuracy of the estmate. 7 Evaluaton Ths secton presents a thorough evaluaton of the proposed method. We evaluate three dfferent aspects of the method: ts accuracy wth respect to varyng scenes, the mpact of the calbraton data and ts applcablty to dfferent geometrc prmtves. We use planar segments n our accuracy study, as calbraton patterns allow a hghly accurate estmaton of the reference planes for quanttatve evaluaton. In ths study, we compare our method to the results of a leastsquares fttng method for planes. In these experments, we nvestgate the nfluence of the dstance between camera and plane, the mpact of the sze of the refned plane, the robustness wth respect to texture and the

9 Geometrc Prmtve Refnement for Structured Lght Cameras 9 a) Angle error b) Dstance error Fg. 6: Normal angle and dstance parameter errors. The red plot shows the error before refnement, black the error after refnement. Crosses and dots show the ndvdual results. The shaded area reflects the standard devaton. Wth the proposed method, the plane parameters can be refned effectvely. mpact of the maxmum number of teratons for the refnement. The second part of the evaluaton contans a study of dfferent aspects of the calbraton procedure. In ths context, we evaluate the mpact of the number of calbraton mages that are used for calbratng the system. Addtonally, we nvestgate how well the calbraton results generalze wth respect to dstances whch have not been covered durng the calbraton. Fnally, we evaluate the proposed method for two other prmtves to demonstrate ts applcablty for geometrc models other than planes. In these examples, ntal estmates of spheres and cubods are refned such that both ther poston and model parameters become more accurate. In ths evaluaton, an Orbbec Astra structured lght camera s used. The camera comes wth a generc factory calbraton whch s used for all camera models, ndependent of ndvdual model varatons. For achevng the hghest possble accuracy, we have estmated the ntrnsc parameters for the specfc camera usng [28] and [19]. 7.1 Evaluaton Data Sets and Expermental Setup The system s calbrated wth a set of 25 calbraton mages that show a whte wall at dstances rangng from 0.7 m to 1.7 m. The wall s approxmately perpendcular to the optcal axs. Accurate reference planes are calculated from checkerboard patterns whch are removed before capturng the dot pattern. Four dfferent data sets wth dfferent scenes are used n ths evaluaton. The frst dataset s captured smlarly as the calbraton data and conssts of 70 mages of a whte wall at dstances rangng from 0.65 m to 1.75 m. We denote ths data set as the wall data set. The ndvdual mages are evenly spread throughout the evaluated dstance range. For each mage the ground truth plane s estmated wth a checkerboard. Before capturng the dot pattern, the checkerboard s removed from the scene. The texture data set conssts of mages that show sx boards wth dfferent surface propertes. Ths data set s used to nvestgate the robustness of the refnement method wth respect to texture and surface propertes. Two addtonal data sets contan mages of spheres and boxes respectvely. The ball data set contans 40 mages of a whte coated ball wth a radus of 10.5 cm from dfferent perspectves and at dfferent dstances. The cubod data set contans 40 mages of a wooden box wth 32.4 cm 26.4 cm 10 cm. The box s captured at varous poses, such that dfferent faces pont towards the camera. The ntal detecton methods for spheres and planes are descrbed n Secton 7.8 and Secton 7.9. In the case of planes, we evaluate results by comparng the plane parameters wth the ground truth planes before and after refnement. Note that the ground truth planes are calculated from checkerboards, and thus are accurate only up to some uncertanty. However, prelmnary results showed that the uncertantes of the checkerboard-based plane estmates are at least 5 tmes smaller than the remanng errors of our refnement results. We use the followng notaton to descrbe the results: the angle between the drectons of the evaluated normal and ground truth normal s denoted by α. The dfference between the dstance parameter of the evaluated and ground truth plane s represented by d. The subscrpts and r refer to the ntal and the refned values. Durng optmzaton, no nformaton from the checkerboard s used. The ntal plane estmates are obtaned wth the probablstc plane segmentaton by [12]. Ths method

10 10 Peter Fuersattel et al. segments the scene nto ts ndvdual planar segments. The segmentaton method calculates a least squares ft for each planar segment, as well as a mask that delmts the partcular segment. We use the proposed method to refne the ntal estmate of the plane model. In ths step, only vald pxels of the segment s mask are consdered. Durng optmzaton, observed and generated vrtual mages are normalzed pxel-wse by ther mean wndow-sze 11 pxels). By default, up to 60 teratons are performed durng optmzaton wth gradent descent. Optmzaton termnates early when the mean squared error for all optmzed pxels changes by less than 10 5 durng one teraton. 7.2 Dstance Between Plane and Camera The frst experment ams at evaluatng the mpact of the dstance between the camera and the plane. In ths evaluaton we analyze the devatons of the normal vector and the dstance parameter for both the ntalzaton and the refned model. We use the wall data set for ths experment. The mages are subdvded nto 10 cm wde bns wth respect to the dstance from the center pxel. In the resultng Fgure 6, we show the mean devaton from the reference normal as an angle and the dfference between the dstance parameters averaged over all planes that belong to a specfc bn. The results show that by usng the proposed method, the error of the least squares plane estmate can be reduced across almost the complete range. The dstance parameter s always mproved by more than 30 %. In the close-range, our method fals to mprove the normal drecton and returns slghtly worse normals than the ntalzaton. We explan ths wth the saturaton of the projected pattern. Whenever the camera gets very close to the wall, and thus to the lmts of ts operatonal range, the spatal extent of the ndvdual dots of the pattern grows untl they eventually merge. Once the camera has a certan dstance > 0.8 m) to the wall, the proposed method relably mproves the ntal drecton of the normal. As the dstance ncreases, mprovements by 60 % and more are possble > 1.6 m). Fg. 7: Average error reducton for dfferent segment szes. Ths plot shows the accuracy gans for the normal and the dstance for segments whch cover a specfc area of the total mage. refnement. We study ths relatonshp wth the mages of the wall data set. We teratvely ncrease the segment sze by 5 % startng from 5 % of the complete mage regon. All segments are postoned n the mage center. Fgure 7 shows the average gans across all mages. For the whole range of segment szes, the accuracy of the dstance parameter of the ntal plane models can be mproved by 40 %. Assumng a correct normal angle, a wrong dstance parameter would evenly affect all evaluated pxels due to wrong dsparty values. In contrast, the drecton of the normal affects all pxels of the plane dfferently. Ths effect can be notced best at the border regons of the segments, and wll be even more dstnct f these regons cover large portons of the mage. Ths explans why the accuracy gans for the normal drecton decrease for smaller segments. Segments wth at least 40 % of the mage sze stll allow mprovements of 30 %, whle t s barely possble to refne segments that cover less than 25 % of the mage. Another nterestng observaton can be made from ths plot: for segments that cover 50 % to 60 % of the mage regon, the best accuraces can be computed. A possble explanaton for ths effect s nhomogeneous scene llumnaton of the projector. Followng ths argumentaton, we thnk that the central 50 % to 60 % of the mage s llumnated best, and thus offers more nformaton for optmzaton than the outer mage regons. 7.3 Influence of the Sze of the Plane Segment Another nterestng characterstc s the relatonshp between accuracy and segment sze. The proposed method explots the nformaton of all pxels that belong to a plane segment. Therefore, t can be expected that hgher accuraces become possble f more pxels are used durng 7.4 Maxmum Number of Iteratons In the current, smple mplementaton the optmzaton termnates f the preset 60 teratons are reached or f the cost functon does not change by more than These parameters need to be adjusted dependng on the requred refnement accuracy.

11 Geometrc Prmtve Refnement for Structured Lght Cameras 11 Fg. 8: Smlarty metrc vs. number of teratons. Only lttle accuracy gans are possble after more than 30 teratons. Seg. α α r d d r % 0.66 cm 0.35 cm 47 % % 1.02 cm 0.41 cm 60 % % 0.50 cm 0.09 cm 81 % % 0.87 cm 0.74 cm 15 % % 0.85 cm 0.53 cm 37 % % 0.75 cm 0.34 cm 54 % Avg % 0.76 cm 0.41 cm 46 % Table 1: Normal and dstance refnement for textured surfaces. α and d are the devatons from the reference normal and dstance. φ s the rato of the ntal value and the refned value. The followng experment nvestgates the decrease of the average cost functon across the teratons wth the wall data set. We evaluate the results after fxed numbers of teratons, startng wth 0 no refnement), to 60 teratons. The results of ths experment are shown n Fgure 8. The fgure shows how the smlarty metrc, n our mplementaton the mean squared dfference of ntenstes, decreases wth addtonal teratons. No large gans n accuracy can be observed after more than 30 teratons. et al. [12], can be refned wth the proposed method. Segment 3 s, due to ts strong texture, one of the more challengng examples n ths evaluaton. Even n ths case the plane s normal drecton and dstance can be refned by 31 % and 81 % respectvely. Addtonally huge accuracy gans for the dstance parameter are possble. Another challengng example s the hghly specular black segment segment 2) for whch the devaton from the reference plane can be reduced by 17 % for the normal drecton and 60 % for the dstance parameter. On average, the drecton of the normal can be mproved by 24 % and the dstance parameter by even 44 %. 7.6 Number of Calbraton Images The number of requred calbraton mages drectly affects the calbraton effort. In ths experment, we order the calbraton mages see Secton 7.1) by ther dstance at the center pxel. From ths lst, N mages are sampled unformly to cover the complete calbraton range. Each of the resultng sets s used as nput to the proposed calbraton method. In ths experment, all mages of the wall data set are refned wth dfferent calbratons. For each calbraton, we calculate the average of all fnal cost functon values to measure how well the fnal vrtual mage matches the observed mage. Fgure 10 shows the results of ths experment for N = {2, 3, 5, 10, 15, 25} calbraton mages. As expected, more calbraton mages wll also ncrease the overall accuracy of the algorthm. However, t s also vald to trade accuracy for the amount of effort one would lke to nvest durng calbraton. Even wth only a few mages, for example less than ten, consderable mprovements are possble. Wth a metrc value of for the ntalzatons, gans of more than 40 % are already possble wth only fve calbraton mages and almost 50 % wth 25 mages. 7.5 Performance on Textured Surfaces In ths secton the robustness wth respect to surface propertes s evaluated. In ths example, the texture data set s used. The scene, shown n Fgure 9, conssts of sx dfferent boards wth dfferent, challengng surfaces, whch are placed on a table. Table 1 shows the results of ths experment. The reference plane of the planar segments s calculated wth a checkerboard pattern ahead of the experment. For all planar segments the least-squares plane ft parameters, that are returned by the method of Holz 7.7 Importance of Calbratng the Whole Range of Operaton Goal of ths experment s to nvestgate the possblty to get refned plane parameters at dstances whch are not covered durng calbraton. Ths s an mportant queston as t also drectly affects the calbraton effort and the re-usablty of calbraton data. For ths evaluaton, we reused the ordered lst of calbraton mages of Secton 7.6. From ths lst, we draw the frst ten mages for calbraton, whch results n a range from 0.70 m to 1 m. Ths calbraton s used to refne the planes of the wall data set.

12 12 Peter Fuersattel et al. a) Color photo b) Infrared mage c) Z-Image d) Segments Fg. 9: Scene wth sx dfferently textured plane segments. Fgure a) shows a top-down color photography of the scene, b) shows the nfrared mage captured by the camera, c) shows the correspondng z-mage dstances n meters) and d) shows the plane segments whch are determned durng ntalzaton. The results for the ndvdual segments are gven n Table 1. The segments 1,3,5 and 6 have a low specularty, segment 4 s moderately specular and segment 2 has hgh specularty. 7.8 Evaluaton on Spheres Ths experment evaluates the performance of the proposed method wth respect to refnng the poston and the radus of spheres. Spheres are represented n terms of a center pont and a radus, thus θ conssts of four unknowns: cx, cy, cz, r). Fg. 10: Mean squared ntensty dfferences for dfferent numbers of calbraton mages. For reference: the ntal mean squared error, wthout refnement, equals In Fgure 11 we compare the resduals for two dfferent set of calbraton mages: frst all calbraton mages and, second the close range subset. The mpact of the lmted calbraton range s not very severe, but can be observed nonetheless. The error of the normal drecton ncreases for planes that are more dstant than 1.2 m as shown n Fgure 11a. The dstance parameter Fgure 11b) s very stable, leadng to barely notceable dfferences between the two calbraton sets. For calbraton wth the close range subset, one could even argue that the dstance parameter, gets even more accurate wthn the range of the calbraton data. We hypothesze that the dot pattern, whch has been derved durng calbraton, resembles the observed pattern n ths range best. In contrast, the other dot pattern wll lkely reflect the observed patterns of the complete calbraton range. The ntal spheres have been estmated wth RANSAC wth a dstance threshold of 2.5 mm. Usng smaller thresholds for RANSAC results n less robust ntal estmates as the dscretzaton error of the camera s larger than a mllmeter. The 3-D ponts whch were ncluded n the estmaton process have been obtaned from a tghtly selected rectangular regon of nterest around the sphere. Furthermore, all pxels whch belong to the background plane have been dentfed wth the plane segmentaton method by Holz et al. [12] and excluded from the estmaton process, thus resultng n a set of ponts wth only very few outlers. Table 2 reports the expermental results for the spheres data set. In the table, we lst the absolute dfference between the true radus of the sphere and the radus of the respectve estmate. The average absolute error of the radus accounts for almost a mllmeter for the RANSAC estmate. Wth the proposed refnement method, ths error can be reduced by more than 50 % to 0.46 mm. Fgure 12 shows four examples taken from the data set. For these examples, the projector has been temporarly dsabled for capturng an mage wthout the overlayng dot pattern. In all cases, the ntal RANSAC estmates are mproved such that the resultng spheres match the observed spheres more closely. The nose, whch s observable n both examples, stems from the low llumnaton n the nfrared spectrum whenever the emtter s dsabled.

13 Geometrc Prmtve Refnement for Structured Lght Cameras 13 a) Angle error b) Dstance error Fg. 11: Normal angle and dstance parameter errors for a subset of calbraton mages. Only the 5 mages wth the smallest dstance between camera and wall have been used for calbraton. The results of the subset are shown n blue, the results of all calbraton mages red) are shown for reference. Stars and dots are used to show ndvdual results. a) b) c) d) Fg. 12: Qualtatve examples from the spheres data set. The green overlays represent the dfference between the mask of the ntal detecton and the refned sphere. In all examples the ntal sphere poston and radus have been refned to a more accurate parameters whch truly match the observed mage. Method RANSAC RANSAC + Proposed Mean Abs. Err. + Std.Dev ± mm ± mm Table 2: Errors for ntal sphere detecton and remanng errors after refnement. 7.9 Evaluaton on Cubods Cubods are another nterestng prmtve that can be refned wth the proposed method. In ths evaluaton we represent a cubod by ts dmensons length wdth heght) and ts poston n terms of a rgd body transformaton that relates the cubod s coordnate system wth the coordnate system of the camera. Consequently, θ contans nne parameters that need to be refned. The method s evaluated on the cubod data set. Fgure 13 shows four selected examples of ths data set. The fgures show the contours of the ntal and refned cubod models n blue and green respectvely. The ntal cubod s found n a three step process: frst the top plane and floor plane are detected. Second, the dmensons of the cubod are obtaned. In the last step, the rotaton and translaton between the cubod and the camera s calculated. The top plane s calculated wth the plane segmentaton method by Holz et al. [12], ntalzed wth a seed coordnate on the top of the box. The floor plane s calculated from all ponts, except for those ponts who support the top plane, va RANSAC. For the estmaton we assume that the floor s parallel to the top plane of the box and consequently fx the normal to be dentcal to the top plane s normal. As the floor resembles the majorty of the pont cloud, the floor plane estmates are very accurate. Length and wdth of the box are calculated as the two major dmensons of the mnmum boundng box of all pxels that support the top plane. The heght s obtaned by calculatng the dstance of one of the cubod s top corner coordnates to the floor plane.

14 14 Peter Fuersattel et al. a) b) c) d) Fg. 13: Qualtatve examples from cubod refnement from the cubod data set. The contours of the ntal and refned cubod are outlned n blue and green respectvely. The examples show that a more accurate ntalzaton leads to more accurate results. The box dmensons and postons n a) and b) are refned notceably. The refnement result n c) shows an mprovement from the ntal detecton, but stll has a small offset at the left contour. The ntal estmate n d) has naccurate rotaton parameters whch can only be corrected to some extent. However, the dmensons of the box have been adjusted to more accurate parameters. The cubod s coordnate system s defned such that the orgn s centered n the cubod, wth the coordnate axes algned wth the length, wdth and heght dmensons of the cubod. Ths enables an easy calculaton of the cubod s corner coordnates n ts own coordnate system. The rotaton and translaton s calculated from correspondng corner coordnates wth Horn s method [13]. In ths evaluaton, we measure the error of the cubods detecton as the sum of the absolute dfferences from the true values. The average error of the ntal detectons equals 0.94 cm. After refnement, ths error s reduced by approxmately 30 % to 0.66 cm. Note that ths error metrc does only consder the dmensons of the boxes but not ther poston wth respect to the camera. For the task of dmensonng, the box poston may be only of secondary nterest, however, for localzaton tasks refned spatal nformaton turn out to be equally mportant as the cubod s dmensons Runtme The runtme for refnng a sngle object depends on the number of pxels the object covers, the maxmum number of teratons and the number of parameters that are requred to descrbe the object. Out of these factors, the last has the largest mpact, as calculatng the dervatves numercally requres to render the the object multple tmes n each optmzaton teraton. In our mplementaton we calculate the central dfference, whch results n the generaton of two vrtual mages per parameter. We have evaluated our two varants of our mplementaton on a consumer notebook. The frst varant only uses the Intel I CPU, whereas the second varant generates the vrtual mage on the GPU Nvda 750M) n a smple CUDA mplementaton. The runtme experments have been conducted wth the wall data set, whch conssts of planes that fll the complete feld of vew. As the number of teratons per mage may vary due to early stop crtera n the gradent descent optmzer we only provde measurements for the average tme requred for a sngle teraton. The CPU mplementaton requres on average 340 ms per teraton, whch nvolves the generaton of nne vrtual mages eght for the central dfferences of the the four parameters, plus one mage for evaluatng the updated parameters). In ths mplementaton we use the complete renderng ppelne, wthout any approxmatons lke a plane-to-plane mappng based on homographes see Secton 4.4). By movng the renderng process to the GPU, the runtme for a sngle teraton could be reduced to 40 ms, resultng n a speedup of more than 8. We beleve that the runtme can be reduced even further f all features of the GPU would be exploted nterpolaton and texture memory). An addtonal speedup could be acheved by portng the complete optmzaton to the GPU. 8 Concluson In ths paper we have presented a novel method for refnng model parameters for geometrc prmtves by explotng the dot pattern of structured lght sensors. The method explots that knowng the exact parameters of a geometrc prmtve n the scene allows to re-render the prmtve such that the rendered mage s dentcal to the camera mage of the structured lght sensor. Unlke regular fttng methods, whch work