Optimized stereo reconstruction of free-form space curves based on a nonuniform rational B-spline model

Transcription

1 1746 J. Opt. Soc. A. A/ Vol. 22, No. 9/ Septeber 2005 Y. J. Xiao and Y. F. Li Optiized stereo reconstruction of free-for space curves based on a nonunifor rational B-spline odel Yi Jun Xiao Departent of Coputing Science, University of Glasgow, Glasgow G12 8QQ, UK Y. F. Li Departent of Manufacturing Engineering and Engineering Manageent, City University of Hong Kong, Kowloon, Hong Kong Received Septeber 24, 2004; revised anuscript received Deceber 20, 2004; accepted January 29, 2005 Analytical reconstruction of 3D curves fro their stereo iages is an iportant issue in coputer vision. We present an optiization fraework for such a proble based on a nonunifor rational B-spline (NURBS) curve odel that converts reconstruction of a 3D curve into reconstruction of control points and weights of a NURBS representation of the curve, accordingly bypassing the error-prone point-to-point correspondence atching. Perspective invariance of NURBS curves and constraints deduced on stereo NURBS curves are eployed to forulate the 3D curve reconstruction proble into a constrained nonlinear optiization. A parallel rectification technique is then adopted to siplify the constraints, and the Levenberg Marquardt algorith is applied to search for the optial solution of the siplified proble. The results fro our experients show that the proposed fraework works stably in the presence of different data saplings, randoly posed noise, and partial loss of data and is potentially suitable for real scenes Optical Society of Aerica OCIS codes: , , , , , INTRODUCTION Traditional stereo reconstruction relies heavily on pointto-point correspondences. Fro a pair of iage points captured in two different views, their corresponding point in 3D space can be reconstructed by the triangulation principle. 1 However, finding point-to-point correspondences in a stereo pair of iages of a real scene has proved very challenging, as the searching space for building such correspondences is extreely large and the searching is usually affected by iage inadequacies such as noise, distortion, lighting variations, etc. Enorous efforts have been devoted to the stereo correspondence proble over the past three decades. Yet the proble is still far fro being solved (see Refs. 2 and 3 for recent surveys) as a result of its intrinsic ill-posed nature. Moreover point-based reconstruction ignores structural inforation between sapling points on object surfaces, thereby raising difficulties in the postprocessing of reconstructed points. In order to avoid the probles with point-based approaches, researchers have applied high-level geoetric priitives to reconstruct 3D scenes. Aong the, 2D priitives such as surface patches 4 6 are suitable for scenes where object surfaces are sooth (with relatively fewer discontinuities), while 1D priitives such as lines 7 10 and curves copleent in environents where these 1D features constitute ajor inforation cues. This paper addresses the case of 1D priitives. The currently used 1D priitives in stereovision are liited to straight lines, 7 10 conics, 11,12 and high-degree algebraic curves. Fro a pair of lines (conics) atched at two different views, their corresponding space lines (conics) can be constructed analytically by intersecting the ray plane (surfaces) passing through these lines (conics). The line (conic) priitives are ore copact copared with sparse points in iages and yield ore robust and efficient atching and reconstruction when applied to artificial scenes where object geoetry has the siple for that can be well described by straight lines and conic segents. However, the line (conic) faily cannot accoodate free-for curves that are anifest in our natural environent (e.g., the for of soe biological objects) and even in an-ade scenes (e.g., edges of soe handcrafts). In order to reconstruct such ore-coplicated shapes by stereovision, soe researchers have eployed highdegree algebraic curves 13,14 as priitives. However, dealing with high-degree algebraic equations has proved error-prone, aking the ethods difficult to be applied practically, and the adopted high-degree algebraic curves in those ethods have been liited to planar representations. So the reconstruction of true 3D (space) curves in free-for shape reains an open question. On the other hand researchers have studied the issues of atching curves in stereo iages, and ore generally fro ulti-views, where the shape of iage curves is considered to be arbitrary rather than confined to a specific known for. As an iage curve contains ore geoetric and structural inforation than a point, and there are fewer curves in an iage than points, atching of curves can be ore robust and efficient. A re /05/ /$ Optical Society of Aerica

2 Y. J. Xiao and Y. F. Li Vol. 22, No. 9/Septeber 2005/ J. Opt. Soc. A. A 1747 cent paper 22 has reported an ipressive rate of correct atching of iage curves of up to 98% in natural scenes. The research work on curve atching so far has certainly advanced the state of the art of extracting inforation in iage curves captured in a 3D scene. At the sae tie a gap arises between curve atching and reconstruction. Since curves can be atched up in stereo iages, the reconstruction of these curves sees a natural step to follow as is done with atched points. Nevertheless, the current ethods based on siple priitives such as points, lines, conics, and high-degree algebraic curves seeingly provide no satisfactory solutions to this proble. Actually 3D reconstruction fro atched curves appears nontrivial in real scenes. If we adopt iage points as the priitives for reconstruction, we still have to find point-to-point correspondences on the iage curves at different views. Even though the point-to-point correspondence atching is sipler when carried out on iage curves (in Ref. 25 it was done over iage sequences), it reains an error-prone proble that degrades the reconstruction quality. First, because points foring an iage curve are projections of 3D points sparsely sapled on the corresponding 3D curve (an iage curve is therefore a projection of a sapling on the original curve), and because two iage curves that signify the sae 3D curve at two different views are not necessarily the sae sapling, the task becoes rather challenging to retrieve in two iage curves the exact corresponding points cast fro the sae physical points in 3D space, even if the iage curves are extracted free of iage inadequacies. Second in practice, noise introduced in iages ight disturb the extracted iage curves fro their proper locations. Such noise and soe other iage inadequacies like selfocclusions and lighting variations ight cause iscapture of soe parts of a 3D curve in its corresponding 2D iage curves, unpredictably underining the point-topoint correspondence-atching process. The line (conic) based ethods suffer siilarly the correspondence proble in reconstructing 3D free-for curves. To establish line (conic) segent correspondences on a pair of iage curves at two views, one ust decopose each iage curve into a set of connected line (conic) segents with one segent on one curve uniquely corresponding to another segent on the other curve. Such decoposition is rather difficult because it iplies that the joint points of the line (conic) segents on one iage curve ust be the correspondences of the joint points on the other curve. The process of establishing such joint-tojoint correspondences is the sae as building point-topoint correspondences on iage curves. Therefore, it has been claied that line-based ethods are not suitable for reconstruction of curved objects, 15 nor are conics intrinsically. To overcoe the drawbacks of reconstruction ethods based on the siple priitives, attepts have been ade in reconstructing a 3D free-for curve as a whole fro its stereo projections (iage curves) where B-spline, a paraetric representation of shapes, was applied in the curve odeling. 26 The results deonstrated the possibility of using spline representation to reconstruct 3D free-for curves in stereovision. However, the caera odel in that work was assued to be affine rather than perspective, liiting its application range. Furtherore, the iage B-spline curves were constructed using standard leastsquares fitting 27 with natural chord paraeterization, resulting in difference between the two paraeterizations of a stereo pair of corresponding iage curves and thereby inducing errors in the final 3D reconstruction. These weaknesses theoretically degrade the applicability of the approach to stereovision. The ai of this paper is to apply the approach to a ore general case: the perspective caera. To this end, nonunifor rational B-spline (NURBS) is adopted as the underlying curve odel. As it is perspective-invariant, this odel akes it possible to accoodate the perspective caera odel in the stereo reconstruction schee. 26 The proble with data paraeterization of curves is tackled by forulating the schee of 3D curve reconstruction fro its stereo projections (iage curves) into an optiization fraework where both the intrinsic paraeters of NURBS curves (see Subsection 2.A) and data paraeterizations are optiized. The iterative algorith of the optiization autoatically leads to the NURBS representations for iage curves on the optial sapling paraeters for data points of these iage curves. The 3D NURBS curve can then be fored by reconstructing its control points and the corresponding weights fro the obtained control points and weights of the 2D NURBS curves. NURBS is a superset of B-spline, therefore inheriting all the good properties of B-spline and yet providing ore flexibility in representing coplex shapes. The reainder of this paper is organized as follows. Section 2 introduces the NURBS curve odel. The perspective invariance of NURBS curves is reinterpreted in an algebraic anner copatible with the algebraic for of caera geoetry. The constraints on the pair of 2D projected curves of a 3D NURBS curve are deduced. Based on the perspective invariance and the deducted constraints, in Section 3 an optiization fraework is established in order to obtain the optial NURBS estiation of 2D iage curves that represent projections of a 3D curve. The siplification of the optiization foralis and the derivative-driven iterative solution to the proble are discussed. The forulas to copute the 3D control points and corresponding weights fro 2D NURBS curves are presented. Experiental results and their quantitative analysis are described in Section 4, followed by conclusions in Section NONUNIFORM RATIONAL B-SPLINE CURVE MODEL The priary goal of the research presented in this paper is to reconstruct 3D free-for curves fro their stereo iages. For odeling free-for curves, NURBS (B-spline is included in the NURBS faily) ethods have played an iportant role in coputer-aided design and coputer graphics because of the any properties of NURBS superior to other shape representations. 28 In coputer vision, the strengths of NURBS have also been recognized in applications on shape recognition, tracking, and atching. 15,29,30 For the work on 3D curve reconstruction in our research, NURBS offers particular advantages suarized as follows:

3 1748 J. Opt. Soc. A. A/ Vol. 22, No. 9/ Septeber 2005 Y. J. Xiao and Y. F. Li 1. A unified curve representation: NURBS can accurately express both free-for and siple algebraic curves, 31 accordingly reducing the representational load in the vision syste and enlarging the application range of NURBS-based approaches. Moreover NURBS is capable of odeling curves in both 2D and 3D, which is iportant in our stereo reconstruction syste where both 3D and 2D curves are involved. 2. Soothness and continuity: A NURBS curve can be treated as a single unit with actually a sooth concatenation of curve segents, which offers better soothness and continuity than polygonal and piecewise-conic representations. Such a property perits analytical coputation of curve derivatives everywhere, providing a potential to apply derivative-based operations to curves, e.g., the iterative optiization for 3D reconstruction given in Section 3 of this paper. 3. Geoetric invariance: A NURBS curve reains NURBS under rigid, affine, or perspective transforation. This allows NURBS to be a universal representation in different coordinate fraes such as world reference frae, object frae, caera frae, and iage frae in which an object geoetry often needs to be transfored fro one to another in vision applications. Indeed, such invariance inspired us to eploy NURBS as the curve representation in our schee for 3D reconstruction fro iage curves that will be reported below. A. Definition of the NURBS Curve Originated fro the rational Bezier equation, the NURBS curve is a generalized extension of B-spline that has the for of vector-valued, piecewise, rational polynoial functions: Ct = W i V i B i,k t W i B i,k t. Here W i is the weight of the ith control point V i, and B i,k t,,1,... are the noralized B-spline basis functions of degree k defined recursively as B i,k t = B i,0 t =1 if u i t u i+1 =0 otherwise, t u i B i,k 1 t + u i+k+1 t B i+1,k 1 t. u i+k u i u i+k+1 u i+1 In Eqs. (2) u i are so-called knots foring a knot vector U=u 0,u 1,..., u +k+1, and t denotes the independent variable for the basis functions. The curve defined in Eq. (1) can be rewritten in the following equivalent for for the sake of siplicity: Ct = V i R i,k t, 1 2 The NURBS for in Eq. (3) is siilar to that of B-spline, except that the rational basis functions R i,k t take the place of B-spline basis functions B i,k t. The rational basis functions are generalizations of nonrational B-spline basis functions, inheriting entirely the analytical properties of B-spline such as differentiability, locality, partition of unity, etc. Furtherore, such generalizations yield ore flexibility in odeling shapes, which not only provides ore options to shape designers but results in soe further iportant properties of the NURBS on its own, e.g., the perspective invariance of NURBS curves (as explained in Subsection 2.B). In this work, we choose cubic NURBS odels k=3 for two reasons: 1. Cubic NURBS is the one capable of representing nonplanar space curves with the least degree, 2. Cubic NURBS curves are C 2 continuous, eaning that the first-order and the second-order derivative vector for every point on the curves can be coputed analytically, accordingly allowing us to apply derivative-based operation on these curves. Once the type of NURBS is fixed, a NURBS curve is deterined only by its control points and weights. Therefore we call control points and weights the intrinsic paraeters of a NURBS curve, distinguishing the fro paraeter t in paraetric equations (1) and (3), which is assigned to a point on the curve in order to calculate its coordinate values. Hereafter we use both Ct and CV i,w i,t to denote a NURBS curve at our convenience. The latter expression is used when the intrinsic paraeters are involved. B. Geoetric Invariance of NURBS Curve Aong any fascinating properties of NURBS representation, geoetric invariance fors the core of our reconstruction fraework. Geoetric invariance allows a NURBS curve to preserve the for of NURBS under a certain geoetric transforation, e.g., rigid, affine, or perspective. Reference 30 has presented an algebraic proof of affine invariance and a geoetric interpretation of perspective invariance of a NURBS curve in which the perspective transforation is defined as a pure central projection. To organize these invariant properties into a unified representational fraework, we reinterpret the perspective invariance of NURBS curve in an algebraic anner, coplying with the algebraic for of caera geoetry and stereo reconstruction. 1 First let us review the affine invariance of NURBS curves using the theore presented in Ref. 30. Theore 1: Affine invariance Suppose Ax+t A represents an affine transforation for a point x; the affine iage of a NURBS curve Ct is a new NURBS curve Ct of the for Ct = i V i R i,k t, R i,k t = W i B i,k t W j B j,k t, j=0 where R i,k t,,1,... are tered rational basis functions. 3 V i = AV i + t A, where A denotes a linear transforation atrix and t A represents a translation vector. (See proof in Ref. 30.) Theore 1 says that affine transforing a NURBS curve can be achieved by affine transforing its control 4

4 Y. J. Xiao and Y. F. Li Vol. 22, No. 9/Septeber 2005/ J. Opt. Soc. A. A 1749 points the transfored curve Ct is a NURBS curve with new control points V i that are affine iages of the original control points V i. Having included rigid (Euclidean) transforation, affine transforation is a apping concerning linear operations in the sae diensionality. Such transforation is often used to odel iage transforations, e.g., the transforation fro caera retina to iage plane, or to odel object transforation in 3D between a world coordinate frae and an object-centered coordinate frae. However, in soe special cases, e.g., when the caera lens is far away fro the object and the object is nearly parallel to the caera retina, the affine transforation can be used to approxiate a 3D 2D perspective projection with acceptable accuracy. 26,32 In those scenarios, the affine caera odel often siplifies the coputation involved in certain tasks such as the work deonstrated in B-spline-based curve reconstruction. 26 Since NURBS is invariant under affine transforation and central projection, 30 the projection of a 3D NURBS curve ust be a 2D NURBS curve, assuing the caera is a pinhole that consists of a central projection and several affine transforations 1 where the nonlinear distortion is ignored. This fact can be interpreted in an algebraic anner coplying with the algebraic fraework of caera geoetry. Let T denote such a perspective projection of a pinhole caera, X=X YZ T denote the coordinate vector of a 3D point, x=x y T denote the coordinate vector of its iage, and let the projection be expressed as TX = x S x 1 =T 1 T 2 T 3 X 1, where T 1, T 2, T 3 are 14 vectors constituting the perspective projection atrix of T, and x 1 T and X 1 T are hoogenous coordinates of x and X. Now we review the perspective invariance of NURBS curves. Theore 2: Perspective invariance Let ct denote the projected curve of a space NURBS curve CV i,w i,t under perspective projection T ; then ct can be expressed in the for ct = where v i =TV i and w i v i B i,k t w i B i,k t, w i = W i T 3 V i 1 The proof of Theore 2 is given in the Appendix. Theore 2 reveals that the original NURBS curve and the projected curve are related to each other by their control points and the corresponding weights; perspectively transforing a NURBS curve is equivalent to perspectively transforing its control points and operating the relevant weights, which are intrinsic paraeters of the NURBS curve. Therefore we can treat the projection of a NURBS curve as a apping in its intrinsic paraeter space without calculating each point on the NURBS curve individually. C. Constraints on Stereo Projections of a NURBS Curve It is well known that stereo projections of a point in 3D satisfy the epipolar constraint. Now that a NURBS curve can be treated as a vector in its intrinsic paraeter space, when a 3D NURBS curve is captured by two caeras at different views, the paraeter vectors of two projected curves will siilarly follow soe constraints. Let T L and T R denote the perspective projections of the left and right caera, respectively; the following constraints can be deduced: 1. Epipolar constraint on control points of projected NURBS curves: Since control points of projected NURBS curves are the projections of control points of the 3D NURBS curve, i.e., v L i =T L V i and v R i =T R V i, where V i denote 3D control points and v L i and v R i denote control points of the projected curve on the left and right retina, respectively, then by use of the siilar ethod of deducing epipolar constraint on iage points at binocular view, 1 L the epipolar constraint on the control points v i and v R i can be derived, e.g., in the for v R i RL TF V i L =0, where F RL is the fundaental atrix of the stereo caera geoetry deterined by T L and T R Weight constraint. Using Eq. (7), the following constraint on the weights of the two projected curves can also be derived: w i L /w i R = T 3 L V i 1T 3 R V i STEREO RECONSTRUCTION OF NONUNIFORM RATIONAL B-SPLINE CURVES A. Proble Stateent and Siplification In Section 2, we presented a brief overview of the NURBS curve odel. We also discussed the geoetric invariance of NURBS representation and derived constraints on the projected curves when a 3D NURBS curve is perspectively observed at two views. Geoetric invariance, especially perspective invariance, exists in all priitives used previously in stereo reconstruction, e.g., straight lines, conics, algebraic curves. The geoetric invariance of NURBS naturally inspired us to consider NURBS as a priitive in stereovision. As the perspective transforation of a NURBS curve can be treated as a apping of its intrinsic paraeter vector, the idea is to reconstruct the intrinsic paraeters of a 3D NURBS curve fro those of its stereo-projected curves using a siilar ethod of reconstructing a 3D point fro its stereo iages. This idea is illustrated in Fig. 1, where a 3D curve fraed by control points V i has two perspective iages on left (L) and right (R) retinas that are basically 2D NURBS curves (denote v L i as control points of the left curve and v R i as con- 9

5 1750 J. Opt. Soc. A. A/ Vol. 22, No. 9/ Septeber 2005 Y. J. Xiao and Y. F. Li s j2 :j 2 =1,2,...,n 2 ;V i ;W i : i =0,1,...,, 11 Fig. 1. trol points of the right curve). In a nondegenerate case, i.e., any pair of control points of a space NURBS curve does not share any ray starting fro an optical center of the caera, we can reconstruct the control points of the 3D NURBS curve by triangulation fro the corresponding control points of the projected 2D NURBS curves, as they satisfy epipolar constraints. Afterwards the weights of the 3D NURBS curve can be calculated by or W i = w i L /T 3 L V i 1, W i = w R i /T 3 R V i 1, 10a 10b where w L i and w R L i denote corresponding weights of v i and v R i. To realize such a schee in real applications where iage curves are initially chains of pixels (digital curves), we need to obtain appropriate NURBS representations for those digital iage curves, which ust satisfy the following constraints: i. the types of the left and the right NURBS curve are the sae (as they represent projections of the sae space curve), ii. each control point of the left curve shares an epipolar plane with its corresponding control point of the right curve, iii. the corresponding weights of the left and the right curve satisfy Eq. (6). The reasons for these constraints and the forulation of the proble under the constraints are explained below. Given a digital iage curve on the left retina consisting of n 1 data points p L j1 = p x L j1, p y L j1 T : j 1 =1,2,...,n 1 and a corresponding iage curve on the right retina consisting of n 2 data points p R j2 = p x R j2, p y R j2 T : j 2 =1,2,...,n 2, our task is to estiate a 3D NURBS curve whose stereo projections best fit the two (left and right) iage curves. In the least-squares easure, such a task can be forulated as the following iniization: n 1 in p L j1 T L CV i,w i,t j1 2 j 1 =1 2 n 2 + j p R j2 T R CV i,w i,s j2 2 =1 with respect to Space curve and its binocular perspective projections. t j1 :j 1 =1,2,...,n 1 ; where CV i,w i,ts is the 3D NURBS curve, T L and T R denote the left and right perspective transforations, and t j1 :j 1 =1,2,...,n 1 and s j2 :j 2 =1,2,...,n 2 are two saplings in the NURBS curve paraeter doain associated with data points in the left and right iage curves. The two saplings are not necessarily related. The ter inside in( ) is the so-called energy function. Foralis (11) optiizes two kinds of paraeters: one is the intrinsic paraeters of the 3D NURBS curve, i.e., v i and W i ; the other is the two paraeter saplings each associated with data points in an iage curve. The reason for optiizing paraeter saplings is that we want to achieve the locations for data points of iage curves on the 3D NURBS curve that result in iniu proxiity between the reconstructed curves and the data points, thereby avoiding explicitly atching the data points in pairs, which is neither accurate nor robust. Such foralis proves to be the particular strength over the B-spline-based work, achieving ore accurate and reliable results, as deonstrated in Section 4. Foralis (11) is apparently a large-scale nonlinear optiization. Such a proble would be coputationally prohibitive without analyzing its specificity and choosing ethods suitable for the specificity, although general purpose optiization techniques (such as siulated annealing, genetic prograing, etc.) ight be applied. The following describes our study of the proble s specificity and the corresponding ethod for tackling the proble. First of all, we observed that perspective invariance of NURBS can be utilized to reduce the nonlinearity of the proble. Assuing that the projections of the 3D NURBS curve CV i,w i,ts are cv i L,w i L,t and cv i R,w i R,s on the left and right retinas, respectively, foralis (11) can be rewritten in the for n 1 in p j1 j 1 =1 with respect to n 2 p R j 2 L cv L i,w L i,t j1 2 + j2=1 2 1 cv R i,w R i,s j2 2 t j1 :j 1 =1,2,...,n 1 ; s j2 :j 2 =1,2,...,n 2 ; v i L ;v i R ;w i L ;w i R :i =0,1,...,; subject to i. epipolar constraints on v i L and v i R, i.e., RL v RT i F v i L =0, i =0,1,...,; 1 1 ii. weight constraints on w i L and w i R, i.e., 12

6 Y. J. Xiao and Y. F. Li Vol. 22, No. 9/Septeber 2005/ J. Opt. Soc. A. A 1751 V i V i w L i /w R i = T 3 L 1T 3 R i =0,1,...,. 1, Copared with foralis (11), foralis (12) has a sipler for in which the nonlinear transforations T L and T R vanish. Although the induced constraints i. and ii. are actually the price of reoving T L and T R, the structure of the proble eerges ore clearly, and the constraints can be further siplified. The ajor difficulty in solving foralis (12) lies in the weight constraints. The relation of corresponding weights w L i and w R i depends not only on a known fundaental atrix, but also the unknown 3D control points V i, which are actually the paraeters we are going to estiate. Therefore the chicken egg puzzle is that if we want to obtain V i we need to solve foralis (12); if we want to solve foralis (12) we have to know V i. The key to this puzzle lies in a rectification of iage curve pairs. We discovered that in a particular stereo configuration, naely, parallel configuration, in which the caeras share a coon iage plane, the constraints in foralis (12) can be greatly siplified so that the proble becoes tractable [no chicken egg puzzles (explained below)]. Moreover it has been proved that an arbitrary nondegenerate stereo pair can be transfored to a parallel stereo pair linearly in hoogeneous coordinates. 33 Therefore we can rectify iage curves to a parallel configuration first and study the curves afterwards, where the following relation can be obtained 33 : T 3 L V i 1= T 3 R V i 1. Consequently the weight constraint in Eq. (9) can be siplified to w i L = w i R 13 Moreover in a parallel stereo configuration, the epipolar constraint also becoes siple 33 : v y i L = v y i R 14 where v y L i and v y R i are the y coordinates of v L i and v R i (siilarly, the x coordinates of v L i and v R i are expressed by v x L i and v x R i ). We can then rewrite foralis (12) to the following unconstrained least-squares relation: 2n in 1 +n 2 f i 2, j=1 with respect to where v x i L, v x i R, v y i, i =0,1,..., t i1 :i 1 =1,2,...,n 1, s i2 :i 2 =1,2,...,n 2, 15 f j =p x L j v x L i R i,3 t j, j =1,2,...n 1 p L y j n1 v y i R i,3 t j n1, j = n 1 +1,2,...2n 1 p R x j 2n1 v x R i R i,3 s j 2n1, j =2n 1 +1,2,...2n 2 v y i R i,3 s j 2n1 n 2, j =2n 1 + n 2 +1,2,...2n 1 + n 2 ; p R y j 2n1 n 2 v y i = v y i L = v y i R and R i,3 are the rational basis functions containing weights w i that satisfy w i =w i L =w i R. The weights in NURBS, while providing great flexibility in odeling curves, cause redundancy in the representation. Theoretically there are an infinite nuber of configurations of control points and weights that would result in the sae curve, as a NURBS curve can be considered a projection of a 4D nonrational B-spline curve constructed in a hoogeneous space of control points and weights, which is a any-to-one apping. Therefore, to be able to obtain a unique representation, currently in ost existing CAD systes, the setting of weights is dependent on the preference of end users, although soe researchers have proposed constraints to copute weights for specific purposes. 34 Following the sae rule, we allow an arbitrary configuration of weights (e.g., we applied a unifor weight setting in our experients) in the algorith by leaving weight-setting a choice of the user. Once the weights are set, our algorith will autoatically estiate the reaining paraeters in foralis (15), naely, the coordinates of the control points and the saplings of iage curves. B. Algorith Although foralis (15) reains nonlinear, its for has been greatly siplified. Each coponent in the fitness function is a rational polynoial of a few variables, and the partial derivatives of the coponents with respect to these variables can be coputed analytically. In such a scenario, derivative-based optiization techniques can be

7 1752 J. Opt. Soc. A. A/ Vol. 22, No. 9/ Septeber 2005 Y. J. Xiao and Y. F. Li used to solve the proble. Aong the derivative-based ethods for nonlinear least-squares probles, the Levenberg Marquardt ethod 35 has proved its popularity in various fields through its siplicity and efficiency. The Levenberg Marquardt ethod requires only oneorder partial derivatives and is therefore well suited for our scenario where the one-order derivatives are analytically available. To solve the proble efficiently, we follow the Levenberg Marquardt schee introduced in Ref. 36 as it is ore coputationally attainable than the other variants of the ethod. Such a schee iteratively searches for better solutions of the optiization proble by solving a linear equation constructed fro a Jacobian atrix in each iteration step. For foralis (15) the Jacobian atrix can not only be coputed analytically but has a sparse and siple for that allows efficient solution of the linear equation in the Levenberg Marquardt iteration step. Figure 2 illustrates the pattern of the Jacobian atrix. Obviously ost of the eleents in the atrix are epty (filled with zeros). We label the subatrices that have nonzero eleents 1, 2, 3, 4, 5, 6, 7, and 8 in the figure. Matrices 1, 2, 3, and 4 are diagonal atrices and their diagonal eleents are derivatives with respect to paraeters t j1 and s j2. Denote t = i R i,k t 16 a 1D rational B-spline function. The derivative t/t can be obtained by directly differentiating Eq. (16). If we substitute v x L i, v y i, v R x i for i and t 1,t 2,...t n1, s 1,s 2,...s n2 for t in t/t, the diagonal eleents of atrices 1, 2, 3, and 4 can be obtained as a. Matrix 1: Substitute v L x i for i and t 1,t 2,...t n1 for t, b. Matrix 2: Substitute v y i for i and t 1,t 2,...t n1 for t, Fig. 2. Jacobian atrix of foralis (15). c. Matrix 3: Substitute v x i R for i and s 1,s 2,...s n2 for t, d. Matrix 4: Substitute v y i for i and s 1,s 2,...s n2 for t. Matrices 5,6,7, and 8 are upper-triangular atrices whose nonzero eleents are derivatives with respect to the coordinate variables of the control points v x i L, v y i, v x i R. Fro Eq. (16), we have t i = R i,k t. 17 For atrix 5, 6, 7, and 8, erely by substituting v x L i, v y i, and v x R i for i and t 1,t 2,...t n1, s 1,s 2,...s n2 for t in Eq. (17), we can obtain the values of all the nonzero eleents as e. Matrix 5: Substitute v L x i for i and t 1,t 2,...t n1 for t, f. Matrix 6: Substitute v y i for i and t 1,t 2,...t n1 for t, g. Matrix 7: Substitute v x R i for i and s 1,s 2,...s n2 for t, h. Matrix 8: Substitute y i for i and S 1,S 2,...S n2 for t. Following the version of the Levenberg Marquardt algorith introduced in Ref. 36, the searching of the optial paraeters is an iterative procedure. In each iteration step the increent of paraeters optiized is the solution of the following linear equations Jd T Jd + Id = gd, 18 where d is the vector containing paraeters x i L, y i, x i R, t j1, and S j2, d is its increent, gd is its gradient vector, Jd is the Jacobian atrix illustrated in Fig. 2, and is a coefficient adjusted in each step. Because of the sparsity of the above-entioned Jacobian atrix, the coplexity of the Levenberg Marquardt algorith for our proble is uch less than that with a dense Jacobian atrix. It is not difficult to prove that Jd T Jd+I is a syetric atrix with On 1 +n 2 + nonzero eleents. To be ore exact the nuber of nonzero eleents is 9n 1 +n if each B-spline curve segent contains an identical nuber of data points. Practically the nubers ight be slightly different, but it will not affect the result that Jd T Jd+I has On 1 +n 2 + nonzero eleents. Through On 1 +n 2 + Jacobian transforations, we can convert Jd T Jd+I to a diagonal atrix. Therefore the running tie of the solution of Eqs. (18) will be as On 1 +n 2 +. The Levenberg Marquardt algorith usually requires liited iterations. Thus we can conclude that the coplexity of the searching algorith is On 1 +n 2 + linear. The linear coplexity arises fro the well-defined foralis in relation (15). An iterative process needs an initialization that sets up a starting state for iteration. In our case, assuing that the end points of iage curves have been atched (this can be siply done using epipolar constraints and disparity order constraints when iage curves are atched), we apply the following noralized chordal paraeterization to initialize paraeters t j1 and s j2 in foralis (15):

8 Y. J. Xiao and Y. F. Li Vol. 22, No. 9/Septeber 2005/ J. Opt. Soc. A. A 1753 j 1 = t j1 r=2 = s j2 t 1 = t 0 p L r p L r 1 n 1 p L r p L r 1 r=2 r=2, 19a, j 1 =2,3,...,n 1 s 1 = s 0 j 2, p R r p R r r=2 19b, j n 2 =2,3,...,n 2 2 p R r p R r where t 0, s 0 are sall positive values to avoid coputational singularity. This paraeterization allocates initial paraeters to data points in the left and right iage curves in the sae paraeter region while keeping the continuity of iage curves. Having t j1 and s j2, we can decopose foralis (15) into the following three linear least-squares relations and estiate the other paraeters x i L, y i, x i R by standard linear least-squares techniques. 27 n 1 in y i,,1,..., n 2 + j 2 =1 j 1 =1 p y R j2 p y L j1 n 1 v y i R i,3 t j1 2 v y i R i,3 s j2 2, 20a in p x L j1 L x i,,1,..., j 1 =1 v x L i R i,3 t j1 2, 20b the 2D control points obtained and known weights, we can reconstruct the NURBS representation for the space curve using the ethod illustrated in Fig. 1. The sapling paraeters in the optiization are locations in the spline paraeter doain that correspond to data points providing inforation about reconstruction regions. The reconstruction regions for the left and right iage curve are int j1,axt j1 and ins j2,axs j2, respectively. Inside the reconstruction regions reconstructed curves are interpolated, while outside the reconstruction regions reconstructed curves are extrapolated. 4. ALGORITHM VALIDATION We have ipleented our algorith in MATLAB 5.3 and validated it using both synthetic and real stereo data sets. The purpose of experienting with synthetic data is to deonstrate specifically the strengths of our algorith by siulating iperfectly posed conditions such as different saplings at the two views, noise in the iage curve extraction, and discontinuities in the iage curves. On the other hand, experients with real stereo iages, designed with different object geoetry and using different curve extraction ethods and different iage capturing devices, help us to investigate the suitability and adaptability of our approach in real-world scenes. All objects exained in the experients (both synthetic and real) exhibit true 3D properties. The results have been copared with those of previous ethods when appropriate. A. Experients with Synthetic Data 1. Different Saplings Because of the discrete nature of iage pixels, an iage curve can be treated as a projection of a set of points (a n 2 in p x R j2 R x i,,1,..., j 2 =1 v x R i R i,3 s j c The above initialization techniques provide a rough solution for the optiization, and the iterative searching will refine the solution step by step until a reasonable precision is achieved judged by the difference between results in consecutive steps (in our case, less than 1%). The outcoe of the iteration is an estiate of the control points of the 2D NURBS curves representing the iage curves on binocular retinas and the sapling paraeters assigned to data points in the iage curves. Fro Fig. 3. Siulated space curve. Fig. 4. Siulated stereo projections of the curve in Fig. 3.

9 1754 J. Opt. Soc. A. A/ Vol. 22, No. 9/ Septeber 2005 Y. J. Xiao and Y. F. Li sapling) in 3D on a continuous curve. For a stereo pair of iage curves, unless specifically designed, the left and right iage curves are usually cast fro different saplings on the 3D curve. Therefore precise point-to-point correspondence atching can hardly be achieved in its nature, even without noise in iages. In our first experient, we evaluated our algorith with siulated iage curves whose data points correspond to different saplings on a 3D free-for curve. To this end, we first generated a ground-truth curve in 3D using paraetric equations: t. We then added Gaussian white noise to t and obtained another paraeter set t=t 1,...t 31 such that t i =t i +n0, t, i=1,2,...,31, where n0, t denotes Gaussian noise with zero ean and standard deviation t.we projected the sapling points t onto the left retina and the sapling points t onto the right retina in a virtual parallel stereo configuration, where the projection atrices of left and right caeras were designed as follows: = 2 cos t tx Y = 2 sin t Z =2t +1 t 0,5/4. Such a curve is truly spatial (nonplanar) as illustrated in Fig. 3. We sapled this curve uniforly in the paraeter region by an interval of t int =/24, giving 31 sapling points on the curve. We denoted these paraeters by t =t 1,...t 31 and the corresponding sapling points by Fig. 5. 3D reconstruction fro projections in Fig. 4. (a) Pointbased, (b) NURBS-based. Table 1. Errors of Reconstruction a with Sapling Differences, in Pixels Noise Levels Approach NURBS-based Point-based t =0.1t int t =0.2t int t =0.3t int e 3d = b e 2dl = e 2dr = e 3d = e 2dl = e 2dr = e 3d = e2dl= e2dr= e3d= e2dl= e2dr= e3d= e2dl= e2dr= e3d= e2dl= e2dr= a e 3d, e 2dl, e 2dr denote errors in 3D, left retina, and right retina, respectively. b Each colun vector contains the values of ean, axiu, iniu, and SD in descending order.

10 Y. J. Xiao and Y. F. Li Vol. 22, No. 9/Septeber 2005/J. Opt. Soc. A. A 1755 Fig. 6. Corrupted stereo projections = T of the curve in Fig. 3 and the back projections of the reconstructed curve T =100 L T = , R If the standard deviation of noise t =0, then the point T L t i on the left retina ust be the exact correspondence of the point T R t i on the right retina, because t i and t i are the sae saplings in 3D. When we assign a positive value to t, the sapling difference between two iage curves appears. If we still treat T L t i and T R t i as a stereo pair of corresponding points, an error will occur in the location of the correspondences. Figure 4 illustrates the stereo projections T L t i and T R t i when t =0.3t int. Using the corresponding pairs T L t i and T R t i, i=1,2,...,31, we can reconstruct 31 3D points by triangulation. Figure 5(a) shows a linear interpolation of these reconstructed points, which akes it easy to visualize the recovered shape of the curve. It is shown that the reconstruction result is strongly affected by the sapling difference between the two iage curves. The recovered 3D curve is neither sooth nor accurate. In contrast, with our algorith a uch better result has been achieved as illustrated in Fig. 5(b). The recovered curve is alost the sae as the original one in Fig. 3, with seven control points in its NURBS representation. The quantitative easureents of reconstruction errors, conducted in both 3D and 2D, are listed in Table 1. Since we know the ground-truth curve in 3D, we can copute closest point distances (CPDs) 37 fro the reconstructed curve (in the reconstruction region) to the ground-truth curve, which reflect proxiity of the two curves. The 2D easureent is conducted in a siilar way, i.e., by coputing CPDs fro the projections of the reconstructed curve (in left and right reconstruction regions) to the projections of the ground-truth curve. Table 1 lists four significant statistics of these CPDs, i.e., the ean values, axiu values, iniu values, and standard deviations (SD) under noise levels t =0.1t int, 0.2t int, and 0.3t int. Applying the sae easureents to point-based reconstruction results, we obtained the data listed in the second row of Table 1. Obviously the NURBS-based reconstruction is overwhelingly better than point-based reconstruction in at least two respects. First, in precision, the axia, eans, and standard deviations of NURBS-based reconstruction errors are uch saller than those of pointbased results in both 3D and 2D. Second, in consistency, Fig. 7. Reconstructed curve fro corrupted stereo projections. the NURBS-based ethod achieves a siilar quality of reconstruction results when the saplings of iage curves vary, showing its insensitivity to the sapling difference, whereas the point-based approach produces reconstruction errors clearly associated with the levels of sapling differences. Actually in the NURBS-based approach, optial positions are assigned to all data points in ters of sapling paraeters on the spline, which are not necessarily required to be the sae at left and right views, while the point-based approach relies crucially on the location precision of point correspondences. Therefore NURBS-based reconstruction is truly curve-based, which eans it needs no correspondence and perits sapling differences between data points in iage curves, while the point-based ethod is particularly constrained to the atching accuracy of data points. 2. Noisy Data In the second experient, we validated our algorith using iage curves corrupted by rando noise. Without loss of generality, we used Gaussian white noise again to siulate the rando noise in curve extraction. Sapling 100 points uniforly along each projected curve of the 3D curve in Fig. 3 (this eans that the interval on the curve between any pair of adjacent points is identical), we added 2D Gaussian noise n0, to the coordinate vectors of these points. In Fig. 6 the jagged curves are the corrupted iage curves = T. We can observe any sharp glitches along the iage curves that strongly change the local properties of the curves. If we apply a point-based approach to this kind of data, accurate pointto-point correspondences will be extreely hard to find because of the noise. However, using the NURBS-based

11 1756 J. Opt. Soc. A. A/ Vol. 22, No. 9/ Septeber 2005 Y. J. Xiao and Y. F. Li algorith, we can obtain an acceptable reconstruction result as shown in Fig. 7, where the reconstructed curve is very close to the original 3D curve. The projections of our reconstructed curve are shown in Fig. 6 as sooth curves, which obviously fit well to the iage curves. The quantitative analysis of the reconstruction errors is given in Table 2 using the sae easuring ethod as that in the first experient. On the whole the reconstruction errors are relatively sall copared with the noise levels, as noise is suppressed by the introduction of the curve odel. At the highest noise level = T, the ean of reconstruction errors in 2D is about 0.2 pixels, while the induced noise reaches about 1.0 pixel in each iage axis on average. The quantitative data also Table 2. Errors of Reconstruction a with Corrupted Data, in Pixels Noise Level e 3d e 2dl e 2dr = T b =0.4 =0.6 =0.8 = T 0.6 T 0.8 T 1.0 T a e 3d, e 2dl, e 2dr denote errors in 3D, left retina, and right retina, respectively. b Each colun vector contains the values of ean, axiu, iniu, and SD in descending order. show that the reconstruction errors in 2D exhibit no sign of an apparent increase when stronger noise is induced, indicating the stability of our algorith in randolyposed noisy conditions according to our reconstruction assuption: finding the 3D curve which best fits the 2D iage data. The 3D reconstruction errors appear to increase as noise increases. The reason is that the stronger noise will cause larger stereo abiguity around curve parts parallel to epipolar lines. We will explain the stereo abiguity further in the experients with real stereo iages below. 3. Fragented Curves Fragented curves coonly occur at the early-vision processing stage of real iages as a result of deficiencies of both iage data and curve extraction ethods, which indeed brings considerable difficulty to the reconstruction. As it is based on the NURBS curve odel, our algorith can potentially deal with this proble to a certain extent. Figure 8 illustrates a pair of stereo iages of the 3D curve in Fig. 3 that consist of data points uniforly sapled on the exact projections of the curve with soe parts issing. Each issing part is randoly selected fro an original projection with its length being 1/10 of the whole length of the projected curve. We assue that we still know the order of curve segents in the fragented curves. For instance, we know fragents 1, 2, and 3 in Fig. 8(a) are successive segents of an iage curve. Feeding our algorith with such fragented iage curves, the result is a continuous and sooth curve as shown in Fig. 9, which resebles rather accurately the original 3D curve in Fig. 3. As our algorith reconstructs a curve as a whole and retains the continuity of data points, issing parts in one iage curve can be copensated for by the corresponding parts in the other iage curve. Table 3 lists the quantitative reconstruction errors, which clearly reain quite low in both 3D and 2D cases. We copared the result with that of the B-spline-based approach 26 (in second row, Table 3), which also reconstructs a curve as a whole. It is clearly seen that the reconstruction error level of the NURBS-based ethod is significantly lower than that of the B-spline-based approach, indicating that the NURBS odel represents (interpolates and extrapolates) data points ore accurately than the B-spline approach in the fragented case. The above three experients with synthetic data revealed certain strengths of our ethod: It is not sensitive Fig. 8. Broken stereo projections of the curve in Fig. 3.

12 Y. J. Xiao and Y. F. Li Vol. 22, No. 9/ Septeber 2005/J. Opt. Soc. A. A 1757 to different saplings in stereo iages, it works reasonably well in noisy conditions, and it has the potential for recovering issing parts of easured curves. B. Experients with Real Iages The purpose of experients with real data is to exaine the suitability and adaptability of the approach in realworld scenes. In the experients we have used two groups of real stereo iages acquired fro different subjects in different scenes at different ties using different stereo capturing devices. In the first experient, the iaging device used was a narrow-baseline stereo head consisting of two B/W caeras calibrated with the projection atrices identified as follows: Fig. 9. Reconstructed curve fro data in Fig T =16.2 L , Table 3. Errors of Reconstruction a with Fragented Data, in Pixels Approach e 3d e 2dl e 2dr NURBS-based B-Spline-based b a e 3d, e 2dl, e 2dr denote errors in 3D, left retina, and right retina, respectively. b Each colun vector contains the values of ean, axiu, iniu, and SD in descending order T =16.1 R The objects of interest are the boundaries of the three vanes of a fan odel (Fig. 10), which exhibit true 3D freefor properties suitable for our tests. Each of these three curves was odeled by a NURBS curve with 23 control points fro iage curves extracted using the Canny operator. Figure 11 displays the reconstructed curves at three orthogonal views. Curves 1, 2, and 3 are the 3D contours of the three vanes that appear fro the botto counter clockwise in the stereo iages. The back projections of the reconstructed curves to the binocular retinas are displayed as curves arked by white x s in Fig. 10, where we can clearly see that the reconstructed 3D shapes of the fan-vanes produce projections that fit well to the iage contours of the fan. In sharp contrast, when we applied straight line priitives for 3D reconstruction, we obtained the uch cluttered result shown in Figure 12 at the sae orthogonal views, indicating again the appropri- Fig. 10. Real stereo iages of a fan odel and back projections of the reconstruction result.

13 1758 J. Opt. Soc. A. A/ Vol. 22, No. 9/ Septeber 2005 Y. J. Xiao and Y. F. Li Fig. 11. Reconstructed curves of the fan odel. Fig. 12. Reconstructed line segents of fan odel in Fig. 10. ateness of choosing NURBS as shape representation and coputing a curve as a whole for reconstructing 3D, freefor curved objects. As there is no ground-truth curve in this experient, we easure only quantitative reconstruction errors of the back projections in 2D, listed in Table 4. The easureents are distances between data points in iage curves and their corresponding points on the projected NURBS curves (the spline paraeter for each data point is known after reconstruction). We use the sae four statistics (ean, axiu, iniu, and standard deviation) to represent the distances. In order to prove our hypothesis that the NURBS-based approach is superior to the B-spline-based approach, 26 we conducted the sae experient with the latter approach using the sae quantitative easureents and copared the results of the two approaches. The statistical data in Table 4 reveal the difference between the reconstruction results obtained fro the two approaches. While the NURBS-based ethod yield subpixel level reconstruction errors with the largest error in all six projected curves of less than 1.1 pixels and the average CPD less than 0.22 pixels, the B-spline-based approach produces pixel level reconstruction errors with the largest error as uch as 9.5 pixels. The relatively large reconstruction errors in the B-spline-based approach are due ainly to the nonotiized sapling paraeters and the affine approxiation of perspective transforation. In the NURBS-based approach, on the other hand, the sapling paraeters are optiized and a fully perspective caera odel is adopted, which draatically reduces the reconstruction errors. Careful readers ight notice the sall distortion occurring in the reconstructed curve 1 shown in Figs. 11(b) and 11(c). The distortion is caused by stereo abiguity, which arises when part of an iage curve overlaps an epipolar line [see the horizontal white line just above the botto of Fig. 10(a)]. In such a condition, the variation of a reconstructed 3D curve on an epiolar plane will yield no change in its projections on the binocular retinas. Such an abiguity is due to the nature of triangulation itself, and it can be dealt with by introducing ore constraints in the optiization fraework. In the second experient, the stereo iages in Fig. 13 were taken by a B/W caera ounted on a anipulator s end effector at two different positions at different ties. The projection atrices in such a stereo configuration were calibrated as T = L , = T R The objects of interest were two wires bent in coplex shapes in 3D space. Our purpose was to reconstruct the curves representing the skeletons of the wires. We applied a NURBS curve odel with 13 control points for the short curve and 23 control points for the long curve. The iage curves were extracted using region segentation and skeleton extraction techniques. 38 The reconstructed curves using our approach are shown in Fig. 14, and their back projections to two binocular retinas are displayed as curves arked by white x s in Fig. 13. It is evident that the result visually exhibits a quality siilar to that in the experient with the fan odel. The quantitative reconstruction errors of the back projections of the bentwire objects using our approach and the B-spline-based approach are listed in Table 5 (the short and long curves are labeled as curve 1 and curve 2,

14 Y. J. Xiao and Y. F. Li Vol. 22, No. 9/Septeber 2005/J. Opt. Soc. A. A 1759 Table 4. Reconstruction Errors a of the Fan Model, in Pixels Approach Curve 1 Curve 2 Curve 3 NURBS-based B-Spline based e 2dl = b e 2dr = e 2dl = e2dr= e2dl= e2dr= e 2dl = e2dl= e2dl= e 2dr = e2dr= e2dr= a e 2dl, e 2dr denote errors in left retina and right retina, respectively. b Each colun vector contains the values of ean, axiu, iniu, and SD in descending order. respectively), where the NURBS-based approach still achieves subpixel accuracy copared with the rather large reconstruction errors in the B-spline-based approach, basically reproducing the results of our first experient. It is also worth entioning that in the above two experients with real data, the iaging devices, the stereo configurations, the iage curve extraction ethods, and the shapes and sizes of objects are all different fro each other. Nevertheless, the NURBS-based ethod does not show significant difference in the quality of the results. This fact points up the potential suitability and adaptability of the approach in real applications. Figures 15 and 16 illustrate residual errors in the iterative processes of the above two experients with real data. Obviously the iterative processes converged at very high rates. In a few iteration steps, the resulting residual errors dropped to a very low level and reain virtually unchanged thereafter. This observation reveals that the objective function forulated in our optiization has a sharp slope around the optial solution. The Levenberg Marquardt approach can then quickly follow the slope to find the solution zones. Figure 17 illustrates the change of control points in the optiization for one curve (curve 2 in Fig. 10(a)). The dashed curve is the iage curve, and the circles are control points of the NURBS curve that represent it. The traces of control points are depicted as the short curves in the upper left-hand corner of Fig. 17(b), with the initial positions represented by crosses. Fro the enlarged part of the curve in Fig. 17(a), we can clearly see that the Fig. 13. Stereo iages of bent wire objects and the back projections of the reconstructed curves. (a) Left iage, (b) right iage. Fig. 14. Reconstructed curves of bent wire objects.