On the Geometry of Visual Correspondence

Transcription

1 International Journal o Computer Vision 21(3), (1997) c 1997 Kluwer Academic Publishers. Manuactured in The Netherlands. On the Geometry o Visual Correspondence CORNELIA FERMÜLLER AND YIANNIS ALOIMONOS Computer Vision Laboratory, Center or Automation Research, University o Maryland, College Park, MD ; and Computational Vision and Active Perception Laboratory, Royal Institute o Technology, S Stockholm, Sweden Received March 1, 1994; Revised January 5, 1995; Accepted May 30, 1995 Abstract. Image displacement ields optical low ields, stereo disparity ields, normal low ields due to rigid motion possess a global geometric structure which is independent o the scene in view. Motion vectors o certain lengths and directions are constrained to lie on the imaging surace at particular loci whose location and orm depends solely on the 3D motion parameters. I optical low ields or stereo disparity ields are considered, then equal vectors are shown to lie on conic sections. Similarly, or normal motion ields, equal vectors lie within regions whose boundaries also constitute conics. By studying various properties o these curves and regions and their relationships, a characterization o the structure o rigid motion ields is given. The goal o this paper is to introduce a concept underlying the global structure o image displacement ields. This concept gives rise to various constraints that could orm the basis o algorithms or the recovery o visual inormation rom multiple views. 1. Introduction The recovery o the structure o a scene rom multiple views and the transormation between the views has been studied in the context o several visual tasks. Indeed, the problems o stereo, 3D motion estimation, calibration, obstacle detection, pose estimation or recognition, etc., can be regarded as special instances o the general recovery problem (Bergholm, 1988; Daniilidis, 1992; Daniilidis and Nagel, 1990; Faugeras, 1992; Faugeras et al., 1987; Faugeras, and Papadopoulo, 1993; Gårding et al., 1993; Koenderink and Doorn, 1991; Navab et al., 1993; Nelson and Aloimonos, 1988; Tistarelli and Sandini, 1992; Ullman, 1979; Ullman and Basri, 1991). Approaches to various aspects o this problem that have appeared in the literature seek a solution in two computational steps. First, a description that relates local measurements in multiple views is developed; local descriptors include stereo disparity measurements, motion disparity measurements, motion ields, partial disparity ields such as those along the X- ory-axes, or normal motion ields (the projections o motion ields along the gradient direction) (Anandan and Weiss, 1985; Hildreth, 1984; Horn and Schunck, 1981; Koenderink, 1986; Shulman and Hervé, 1989). Second, knowledge o the model o the geometric transormation between the multiple views provides constraints on the local descriptors; these constraints are used to relate image measurements to the 3D scene and viewing geometry (Faugeras and Maybank, 1990; Horn, 1990; Koenderink and van Doorn, 1991; Liu and Huang, 1988; Maybank, 1993; Prazdny, 1980; Spetsakis and Aloimonos, 1988; Tsai and Huang, 1984). This paper deals with the case where the transormation between the views is described by a rigid motion. The rigid motion model imposes constraints on the local image measurements (disparity measurements, optical low ield, normal low ield), which thus have a certain global structure. The goal o this study is to make explicit aspects o this structure that are due only to rigid motion. In the remainder o this paper we will use the terms motion vector and motion ield to reer to the 2D image displacements (local descriptors), and normal motion vector or normal motion ield to reer to the motion vector components along the image gradients.

2 224 Fermüller and Aloimonos Figure 1. The rigid motion constrains the motion vectors to lie on conic sections in the image plane. C 1, C 2 and C 3 are curves containing the motion vectors o values (0.5, 0.5), (0.1, 0.7), and ( 0.4, 0.4). Figure 2. For a normal motion ield all vectors o certain value are constrained to lie within regions. The boundaries o these regions depend on the 3D motion What is to Come Beore we proceed, in order to provide the reader with an intuitive notion o the the global structure o rigid motion ields, which is the subject o this study, we present in Figs. 1 and 2 two simple examples. Figure 1 shows a motion ield generated by an observer moving rigidly with regard to some surace. The analysis in this paper will show that the rigid motion, independently o the scene in view, constrains the locations o the motion vectors that have certain values. For example, all vectors (u = 0.5, v = 0.5) lie on the conic section C 1. (This does not mean that all vectors on C 1 have motion vector (0.5, 0.5); it means only that i there exists in the image a point with motion vector equal to (0.5, 0.5), this point will lie on the curve C 1.) Similarly, curves C 2 and C 3 contain all points with motion vectors (0.1, 0.7) and ( 0.4, 0.4) respectively. This study investigates properties o the classes o curves which are loci o points where the motion vector has some property. For example, we will show that all curves corresponding to vectors o ixed value intersect in one point whose coordinates encode inormation about the translational component o the motion, i.e., the FOE. Figure 2 shows a normal motion ield (i.e., the component o the motion vector along the image gradient at every point) resulting rom the same rigid motion. Points in the image plane where the normal motion vector can have a particular length and direction are clustered in regions whose boundaries depend on the 3D motion. To illustrate this in Fig. 2 all normal motion vectors parallel to the Y -axis and o length 0.3 are shown to be in the area marked by vertical lines. We will study the relationships o such areas to each other and to the curves described above. For example, we will see that the intersection o the region boundaries provides the parameters o the translation as well as the rotation o the underlying 3D motion. The main goal o this paper is to reveal global geometric aspects o low or displacement ields which are due to rigid motion. In this sense this study is related to a small number o previous studies, which were concerned with a global decomposition o motion ields or the specialized cases o pure translation (Jain, 1983; Horn and Weldon, 1987) and pure rotation (Aloimonos and Brown, 1984). In earlier work (Fermüller, 1993; Fermüller and Aloimonos, 1994) we dealt with the general case and described certain global properties o rigid normal motion ields maniested as patterns deined on the sign o subsets o the motion vectors. In this paper we ormulate a more general ramework having in mind to investigate global structures through the study o unctions o the the values o 2D rigid displacement ields. This is achieved by deining unctions on the vectors o the low ield and studying the location o the level sets o these unctions. The unctions employed here are o a simple nature. In particular, we concentrate on optic low vectors o constant value, normal low vectors o constant value, and on certain linear and quadratic unctions deined on the normal low, and we also relate this analysis to our earlier results. The remainder o this paper is divided in three parts. The irst part (Sections 2 and 3) is devoted to the

3 On the Geometry o Visual Correspondence 225 analysis o rigid motion ields, the second part (Section 4) is concerned with rigid normal motion ields, and in the last part (Section 5) applications to a variety o visual tasks using the theoretical results, are outlined. In particular, Section 2 describes the wellknown equations relating the image motion ield to the 3D motion or the case o a planar retina, and Section 3 explains the structure o the iso-motion curves and gives an analysis o the relationships between dierent iso-motion curves. An iso-motion curve C u is a locus o points where the vector ield representing the image motion could take on a ixed value deined in terms o u. Analogously, when dealing with normal low, we encounter iso-normal motion regions. Section 4 studies the relationship between such regions and the 3D motion parameters. The concept o selecting vectors o certain lengths and directions is extended to certain vector valued unctions. In the section on applications, we discuss how the concepts and structures introduced in the paper could serve as the basis or a variety o perceptual mechanisms underlying visual tasks. The advantage o the constraints developed is due to the act, that they are deined globally on the image, as they relate motion measurements rom dierent parts o the image to each other. The globality gives the constraints the potential o being exploited in tasks related to the recovery o the parameters describing the rigid motion coniguration. They may be used in algorithms computing the extrinsic as well as the intrinsic parameters o a rigid motion coniguration or stereo setting. They also may be exploited to veriy that a vector ield is only due to rigid motion and to locate areas in the image where this constraint does not hold or example, in the detection and localization o independently moving objects or a moving observer. In general, they could be utilized whenever navigational problems are addressed that involve estimating aspects o 3D motion, such as image stabilization, servoing, or docking. Finally, the appendix develops the equivalent o the isomotion contours and the iso-normal motion regions on the sphere, thus providing a simple geometric intuition to these concepts. 2. Localization o Motion Measurements The 2D motion ield on an imaging surace is the projection o the 3D motion ield o the scene points moving relative to that surace. I this motion is rigid, it is composed o a translation t and a rotation ω. For the case o a moving camera in a stationary environment each scene point R = (X, Y, Z) measured with respect to a Figure 3. retina. Image ormation using perspective projection on a planar coordinate system OXYZ ixed to the camera moves relative to the camera with velocity R, where R = t ω R. (1) Projecting the 3D motion vectors on a retina o a given shape gives the image motion ield. I the center o projection is at the origin O and the image is ormed on a plane orthogonal to the Z-axis at distance (ocal length) rom the nodal point (see Fig. 3), the relation between the image point r = (x, y, ) and the scene point R under perspective projection is r = R ẑ R, where ẑ is a unit vector in the direction o the Z-axis and the denotes the inner product o vectors. I we now dierentiate r with respect to time, and substitute or R, we obtain the ollowing equation or r: r = 1 R ẑ ((t ẑ)r t) 1 [rωẑ]r ω r, (2) where [rωẑ] = r (ω ẑ) (triple product). The irst term or r in Eq. (2) denotes the translational motion component which depends on the depth Z = R ẑ, while the second term denotes the rotational component which does not depend on depth, but only on the three rotational parameters. As can be seen rom the equations, using perspective projection, only the scaled translation t can be recovered. The points Z where the axis o translation pierces the retina are called the Focus o Expansion (FOE) in the case o positive

4 226 Fermüller and Aloimonos translation along the Z-axis or Focus o Contraction (FOC) otherwise, since at these points the translational motion components are zero and all translational motion vectors point away rom or towards these points. Similarly, we call the point where ω (the rotation axis) pierces the retina the Axis o Rotation point (AOR). At this point the rotational motion is zero. We are concerned with how the act that the motion is rigid constrains the image motion vector ield. Looking at a single measurement, we see that due to rigidity the motion vector at every point is constrained to lie in a one-dimensional subspace (deined by the rotational component and a translational component o which we know the direction but not the length). Since the distance rom the scene to the image is positive, the possible space or the motion vector at every point is urther reduced to a hal-space. In order to separate the constraints on the motion ield due to shape rom those due to motion, we take the approach o studying answers to the ollowing questions: Given a certain value v(p) or a motion vector at point p (or a vector valued unction), where are the locations p on the retina, or which the motion vector ṙ at p (hereater ṙ(p)) could take the value v(p)? These concepts are explored below. 3. Iso-Motion Contours in the Plane 3.1. Characterization o the Form o Iso-Motion Contours in the Plane I the scene is projected on a planar retina the image velocity ield is given by Eq. (2). We express this equation in the more common component notation: r = (ṙ 1, ṙ 2, ṙ 3 ). ṙ 3 is always zero. We denote ṙ 1 by u and ṙ 2 by v, t = (U, V, W ), ω = (α, β, γ ). Iwe introduce new coordinates or the direction o translation (x 0, y 0 ) = ( U W, V ), we obtain the well-known W equations (Longuet-Higgins and Prazdny, 1980) u = u trans + u rot = ( x 0 + x) W Z + α xy v = v trans + v rot = ( y 0 + y) W ( y 2 Z + α ( x 2 ) β + +γy (3) ) + β xy γx (4) We are interested in the locus o points with motion vector u = (u,v). To obtain the locations (x, y) in the image plane or which the motion vector has some constant value (u,v), we bring the rotational components in Eqs. (3) and (4) to the let side and divide (3) by (4): u u rot = u trans v v rot v trans u ( α xy β ( x 2 + ) +γy ) v ( α ( y 2 + ) β xy γx ) = x x 0 (5) y y 0 which results in the equation φ(x, y,u,v) = y 2 ( αx0 ) +γ ( βx0 xy + αy ) 0 ( ) +x 2 βy0 +γ x(α + γ x 0 v) y(β + γ y 0 + u) + x 0 (α v) + y 0 (β + u) = 0 (6) describing a second order curve in the image plane. Hereater, we will reer to the curves given by Eq. (6) as iso-motion contours, in particular as u iso-motion contours C u (or (u,v) iso-motion contours C (u,v) ) when denoting the parameterization by u ((u,v)). Since the values o u and v do not appear in the quadratic terms o Eq. (6), but only in the linear and constant terms, the nature o the curve (i.e., whether it is an ellipse, hyperbola, or parabola) is independent o u, v, and thus is the same or all such parametrized curves o a given motion ield. The axes o the conics are all parallel to each other with slopes m and 1 m, where m is the positive o the two values (Selby, 1972) (βy 0 αx 0 ) (α 2 + β 2 ) ( x0 2 + ) y2 0. (βx 0 + αy 0 ) The nature o the iso-motion contours depends on the values o the translation and rotation. Depending on the value l, where ( βx0 l = + αy ) 2 ( )( ) 0 αx0 βy0 4 + γ + γ the contour is a hyperbola i l > 0, an ellipse i l < 0, or a parabola i l = 0. Figures 4(a) and (b) show two classes o iso-motion curves or two dierent motion ields.

5 On the Geometry o Visual Correspondence 227 Figure 4. (a) Ellipses as iso-motion contours, (b) hyperbolas as iso-motion contours The Structure o Iso-Motion Contours in the Plane In this section relationships between iso-motion contours in the plane are investigated. We are interested in studying the intersections o iso-motion contours and the relationship between the iso-motion contours and the possible motion measurements along these curves. To characterize the intersections o iso-motion contours, we look at the possible values the motion vector can take at a point. For a given rigid motion the rotational component o the motion vector and the direction o the translational component have ixed values or a speciic point. The only degree o reedom lies in the variability o the depth. At point P(x, y) the motion vector can take the value u(x, y) = u rot (x, y) + λu tr (x, y), (7) where u tr (x, y) = (x x 0, y y 0 ) and λ is the scaled depth value W. The iso-motion contours that pass Z through P(x, y) have to correspond to the motion vectors given by Eq. (7), and thus every point can be considered as the intersection o the amily o u(x, y) isomotion contours, as deined in (7). Let us now consider, as beore, the iso-motion contours o parallel motion vectors, i.e., let us consider the iso-motion contours o values k(u,v), where (u,v)is a unit vector and k a scalar R. These can be obtained rom Eq. (6) by substituting ku or u and kv or v. Equation (6) can be written as a sum o a quadratic expression q(x, y) and a linear expression p(x, y) multiplied by k: φ(x, y,u,v,k) = q(x,y)+kp(x, y) = 0 ( ) ( = y 2 αx0 βx0 + γ xy + αy ) 0 ( ) +x 2 βy0 +γ x(α + γ x 0 ) y(β + γ y 0 ) + x 0 α + y 0 β + k(xv yu x 0 v + y 0 u) = 0 (8) Independently o k, the two intersection points o q(x, y) and p(x, y) are elements o all the (ku,kv) iso-motion contours φ(x, y,u,v,k). Thus the amily o contours has two common intersection points. Both these points, since they lie on φ(x, y,u,v,k) or every k, also lie on the zero motion contour. Thus all amilies φ(x, y,u,v,k)o iso-motion contours intersect at two points on the zero motion contour (see Fig. 5). One o them is the FOE. The other one is denoted by P k(u,v). Since P k(u,v) lies on all φ(x, y,u,v,k) iso-motion contours the value o the motion vector at P k(u,v) will certainly be k(u,v)or some k. Thus the direction o the motion vector or every point on the zero motion contour is deined. Since P k(u,v) lies on the zero motion contour, and thus rom Eq. (7) it ollows that u tr = µu rot (i.e., the translational component is parallel to the rotational one), we ind that the motion vector or every

6 228 Fermüller and Aloimonos Figure 5. The amily φ(x, y,u,v,k) o iso-motion contours intersect at two points on the zero motion contour: The FOE and P k(u,v). Figure 7. For any point P with motion vector v on a general u iso-motion contour the motion vectors v, where v = v u, is parallel to the translational component v Depth Positivity A u iso-motion contour is deined by Eq. (5) as the geometrical locus o points or which the direction o translation is equal to the direction o the dierence between u and the rotational component, not considering any constraints on the depth. Since the scene lies in ront o the camera, and thus an additional restriction is imposed on the values, only a part o the iso-motion contour can contain motion vectors. Let us assume that the translation along the Z-axis has positive sign and thus λ = W > 0. Thereore, rom Z Eq. (7), we obtain two inequalities Figure 6. The motion vector in P k(u,v) is parallel to (u,v). For every point on the zero motion contour the motion vector lies on a line connecting the point with the FOE. point on the zero motion contour is along the direction o its translational component and thus lies on a line connecting the point with the FOE (see Fig. 6). I we look at points on general u motion contours, where u is any 2D vector, we can make the ollowing statement about the direction o the motion vector there. Since any point P is the intersection o isomotion contours o value u + λu tr, i we subtract u rom the actual motion vector v, we are let with a vector v which is parallel to the translational motion (v = v u = λu tr ) and thus lies on a line that passes through the FOE (see Fig. 7). (x x 0 )(u u rot )>0 (9) (y y 0 )(v v rot )>0 (10) These inequalities deine a curve segment whose endpoints are those or which λ = 0 and λ =.Iλ=0, u = u rot,iλ=,u=u trans, which means that the curve segment connects a point R u (or which the rotational motion is equal to u) to the FOE, the irst point corresponding to ininite depth, and the second to zero depth (i.e., a scene point on the image plane). Along the curve segment the depth decreases continuously. This simple observation can give rise to qualitative techniques or the estimation o structure. In Fig. 8 two curves o a amily o k(u,v) iso-motion contours are displayed. The curve segments which correspond to positive depth values are marked with circles.

7 On the Geometry o Visual Correspondence 229 Figure 8. Two o a amily o ku iso-motion contours (C 0 and C 0.5,0.5 ): The curve segments or which the depth is positive are marked by circles. The endpoints o these segments are the FOE and the points R ku Fixating Stereo The image displacement ield due to binocular disparity measurements obtained by a stereo system ixating at a point deserves some separate discussion. This coniguration has oten been studied in the psychophysical literature (von Helmholtz, 1896). In particular, a concept that can be regarded as a special case o the iso-motion contours, has been investigated: The locus o points in space that yield zero disparity, the so-called horopter. For a general stereo system in ixation (see Fig. 9) the horopter consists o two intersecting curves, the irst being a circle in the plane deined by the two optical axes o the two cameras passing through the two nodal points and the ixation point, and the second being orthogonal to the plane o the circle and passing through the ixation point. Let us ix a coordinate system to the let camera and let us describe the disparity (motion) o the right camera with respect to let one. We assume that the baseline is in the XZ-plane. To obtain a uniied notation, we use here the small baseline approximation in order to be able to employ the same dierential equations as beore. Our concern here is the study o the structure o disparity ields, and the structure itsel is not aected by this assumption. The center o the right camera is in the XZ-plane and the rotation, i the two cameras ixate at one point, is only around the Y -axis. Thereore, there are only two unknown motion parameters or a ixating stereo coniguration, namely x 0 and β. We thus obtain the ollowing equation or the iso-motion contours o Figure 9. Binocular viewing geometry under ixation: The dotted circle through the ixation point and the eyes and the dotted line perpendicular to this circle indicate the horopter, i.e., the locus o points in 3D that yield zero disparity. a ixating stereo in the image plane: x 0 βxy xv 1 + y(β + u 1 ) x 0 v 1 = 0 (11) The iso-motion contours are hyperbolas. Their axes are parallel to the medians, and the center o their axes is (Selby, 1972) ( ( β u1 ) (x c, y c ) = βx 0, v ) 1. βx 0 The zero motion contour, which is the projection o the horopter on the image plane, is deined as ( ) x0 x yβ + = 0 which is the equation o two lines, one being the x- axis, the other a line parallel to the y-axis with x- coordinate x = 2 x 0. These lines, o course, constitute a degenerate hyperbola. The amilies o iso-motion contours corresponding to parallel measurements (i.e., the k(u,v)iso-motion contours) have the FOE as one intersection point and as the other a point P k(u,v) which lies on the part o the zero motion contour or which x = 2 x 0. Thus or every point along the line x = 2 x 0

8 230 Fermüller and Aloimonos motion u n is u n = (u n)n. Substituting or the components o u rom (3) and (4), we obtain u n or the value o the vector u n along the gradient direction: and thus u n = u rot n x + u trans n x + v rot n y + v trans n y (12) Figure 10. Iso-motion contours or ixating stereo: For all points along a line parallel to the Y -axis the motion vectors are in the direction o their translational motion components. the direction o the low is in the direction o the translational motion component (see Fig. 10). 4. Normal Motion Constraints The vector ield which represents the components o the motion ield perpendicular to edges is reerred to as the normal motion ield (Aloimonos, 1990; Verri and Poggio, 1989). It is uniquely deined by local image measurements and can be derived without conronting the aperture problem. Although a normal motion ield seems to contain less inormation than the exact motion ield, the motion involved is still maniested in it. In particular, i the normal motion ield is due to a rigid motion, it possesses a certain structure. In this section we investigate the locus o points in the image plane where the normal motion vector can take on a certain value. We irst deal with the case o a normal motion vector o constant value, and later generalize to certain vector-valued unctions linear and quadratic in the image coordinates. It will be shown that the only constraint or the location o these image points originates rom the act that the depth has to be positive. As a result the possible locations are ound to be connected areas in the image plane. The shapes o these areas are deined by the rigid motion Iso-Normal Motion Areas I u is the motion vector at a point (x, y) and n = (n x, n y ) is a unit vector in gradient direction, the normal W Z ((x x 0)n x + (y y 0 )n y ) ( = u n α xy ( x 2 β + ( ( y 2 α + ) β xy γx ) ) +γy n x ) n y. (13) We are concerned with the question: Where in the image plane can the normal motion ield take on a certain constant value u n (i.e., where could normal motion vectors o a certain length u n and direction (n x, n y ) be?)? The depth has to be positive. I we assume W > 0, we obtain the ollowing inequality: [ ( u n α xy ( α ( y 2 ( x 2 β ) ) + +γy n x ) + β xy ] γx )n y [(x x 0 )n x +(y y 0 )n y ] > 0 h(u n,α,β,γ,x,y) g(x 0,y 0,x,y)>0 (14) h(u n,α,β,γ,x,y)=u n (α xy β( x2 + )+γy) n x (α( y2 + ) β xy γx)n y and g(x 0, y 0, x, y) = (x x 0 )n x + (y y 0 )n y. The equation h(x, y) = 0 describes a hyperbola that splits the image plane in an area where h(x, y)>0 and an area where h(x, y)<0. The equation g(x, y) = 0 describes a line through the FOE, which is perpendicular to (n x, n y ), and which separates the plane into an area where g(x, y) is positive and an area where g(x, y) is negative. Thus, through this inequality a region I un consisting o two areas bounded by a hyperbola and a line are deined as the locations where the normal motion could take on a certain value u n. The two areas meet at one point, the intersection o the hyperbola and the line. This point, which contains inormation about the whole pattern, will be denoted by S un (see Fig. 11).

9 On the Geometry o Visual Correspondence 231 Figure 11. Iso-normal motion regions are bounded by a line (g(x, y) = 0) and a hyperbola (h(x, y) = 0). In the other areas o the image plane the value o the motion vectors in direction (n x, n y ) is constrained. Where h(x, y)<0wehaveu n >u rot n (i.e., the rotational component o the normal motion is greater than u n ). Where g(x, y)>0 the translational component o the normal motion is greater than zero. In the area where h(x, y)<0 and g(x, y)>0, we can thus conclude that the normal motion (the sum o the rotational and translational components) is greater than u n. Similarly, where h(x, y)>0 and g(x, y)<0, the value o the normal motion has to be smaller than u n (see Fig. 12). To summarize these results, considering or a given normal motion ield due to rigid motion the vectors along the gradient direction (n x, n y ), we ind that the image plane is split by a hyperbola and a line into our areas. All vectors which are o length u n are in two opposite areas. One o the two other areas contains only values greater than u n, and the other only values smaller than u n. The line (g(x, y) = 0) is deined by the translational motion; it passes through the FOE and is perpendicular to the gradient (n x, n y ) o the normal motion vector. Thereore, this line is described by only one unknown parameter (its direction is known). Furthermore, the line is independent o u n, the value o the normal motion vector. For any general u n the hyperbola (h(u n, x, y)) is deined by the three rotational parameters. For the case when u n = 0, the number o unknowns reduces to two ( α γ and β expressing the direction o the rotation axis). I we consider γ parallel Figure 12. Separation o the image plane into areas by the values o the normal motion vectors in certain directions: In the area marked by horizontal lines all normal motion vectors in direction (n x, n y ) are greater than u n. In the area marked by vertical lines all normal motion vectors in direction (n x, n y ) are smaller than u n. Vectors o length u n in direction (n x, n y ) can only be in the complementary areas (the region I un ). Figure 13. Iso-normal motion regions corresponding to parallel normal motion vectors: The hyperbolas h(u n0, x, y), h(u n1, x, y), and h(u n2, x, y) are corresponding to the parallel normal motion vectors u n0, u n1, and u n2. The length o u n0 is zero, thus h(u n0, x, y) passes through the AOR. The line g(x, y) = 0 is independent o the length o the normal motion vector and thus the same or all parallel normal motion vectors. normal motion vectors, i.e., normal motion vectors o value k(u n,v n ), where k any scalar, we ind areas in the image plane which are bounded by a line that is the same or all values and hyperbolas which dier only in their linear terms (see Fig. 13).

10 232 Fermüller and Aloimonos Figure 14. The intersection points S un o the zero normal motion regions lie on the zero motion contour. Figure 15. The iso-motion curves C u1, C u2, C u3, and C u4, (where u 1 n = u n, u 2 n = u n, u 3 n = u n, u 4 n = u n ) intersect on a line through the FOE perpendicular to n Relation Between Iso-Normal Motion Regions and Iso-Motion Contours The intersection point o the line and the hyperbola S un is a salient point in the description o the iso-normal motion areas. We describe here some relationships between the intersection points and the iso-motion contours. Later these relations will be exploited in the development o algorithms. First, let us consider the zero iso-normal motion areas only. For points on the line g(x, y) = 0 the translational normal motion vector component in the direction o the gradient (n x, n y ) is zero. For points along the hyperbola h(x, y) = 0 the rotational normal motion component in direction (n x, n y ) is zero. Thereore, it ollows that at the intersection point S un the translational motion component is parallel to the rotational motion component, and thus or all zero iso-normal motion areas S un lies on the zero iso-motion contour (see Fig. 14). S un is also element o every k( n y, n x ) iso-motion contour. The ollowing can be derived or the intersection point S un o any general u n iso-normal motion area: Since S un lies on the line g(x, y) = 0, the translational motion component at S un is parallel to the line and perpendicular to u n, and the translational normal motion component along (n x, n y ) is always zero. Thereore, through S un pass all those u iso motion contours or which u n = u n (i.e., all the motion vectors pass whose projection on the gradient direction yields in the same normal motion vector) (see Fig. 15) Bounded Depth Measurements The constraints developed so ar are only due to rigid motion. In most practical applications upper and lower bound estimates o the distance rom the image to the scene, or the scaled distance W, are available. I in Z Eq. (13) we substitute τ min or the minimum value and τ max or the maximum value o W, we obtain two equations o hyperbolas. These equations deine the bound- Z aries o the area in which normal motion vectors o a certain length and direction can be ound. We will reer to these areas as normal motion bands; an illustration is given in Fig. 16. Figure 16. Bounds on the value W Z constrain the normal motion vectors to an area deined by two hyperbolas, the so-called normal motion band.

11 On the Geometry o Visual Correspondence 233 I the upper bound or the depth becomes ininity, the corresponding hyperbola approaches the hyperbola o the iso-normal motion area. Such situations oten occur in outdoor scenes. I the lower bound or the depth becomes zero, the corresponding hyperbola approaches the line. Clearly, the smaller the possible range or the depth estimates, the smaller the bounded area. In particular, i the motion is only rotational, the measurements are only along the hyperbola o the iso-normal motion area. Considering normal motion vectors o length zero, this hyperbola passes through the AOR. I the motion is purely translational, all normal motion measurements o value zero are on the line. An illustration is given in Fig. 17. Figures 17(a), (c) and (e) show synthetically created normal motion ields. The ields in Figs. 17(c) and (e) are due to only translation and only rotation, and the normal motion ield in Fig. 17(a) is due to both these motions. Overlaid over the normal motion ield in 17(a) are vectors showing the gradient directions n 1 and n 2, or which normal motion vectors o length zero have been selected, and the corresponding boundaries o the normal motion areas. In this and any other implementation shown in this paper vectors o direct n and length u n are selected, i they lie within an interval [n + ɛ n, n ɛ n ] and [u n + ɛ un, u n ɛ un ]. In Figs.17(b), (d) and () the areas in the image plane, where normal motion vectors o length zero in direction n 1 and n 2 were ound are marked by black and grey squares. The ollowing can be concluded about the intersection o general iso-normal motion bands: For dierent gradient directions n i we consider the normal motion vectors o length u ni, where u ni is the projection o the same motion vector u (i.e., u ni = u n i ). Any scaled depth value τ deines a curve as the location o normal motion vectors o value u ni. For one τ the curves corresponding to dierent gradient directions intersect in one point. This point lies on the u iso-motion contour. Thus the intersection o all u ni iso-normal motion bands is a curve segment that lies on the u iso-motion contour. In particular, all iso-normal motion areas o value zero intersect on the zero motion contour (see Fig. 18) Coaxis and Copoint Vectors In this section the concept o selection o normal motion vectors o a given length and direction is generalized. Instead o considering vectors o the same value, we examine various classes o vector-valued unctions. In Figure 17. (a) Normal motion ield due to translation and rotation. Superimposed on the vector ield are the boundaries o the two normal motion areas corresponding to the vectors o length zero in direction n 1 and n 2, (b) the black and grey squares denote locations where in the normal motion ield o (a) vectors o length zero in direction n 1 and n 2 were ound, (c) and (d) normal motion ield only due to translation and corresponding normal motion vectors o length zero in direction n 1 and n 2. The intersection o areas corresponding to dierent normal motion vectors o length zero gives the FOE, (e) and () normal motion ield only due to rotation and corresponding normal motion vectors o length zero in direction n 1 and n 2. The intersection o areas corresponding to dierent normal motion vectors o length zero gives the AOR. particular, we investigate the coaxis and copoint vectors, which we described in earlier work (Fermüller, 1993; Fermüller and Aloimonos, 1993). The copoint vectors are deined with respect to a point. The (r, s) copoint vectors are deined as the normal motion vectors which are perpendicular to straight lines passing through the point (r, s). An(r, s)copoint vector at a point (x, y) is parallel to the vector (s y, x r) (see Fig. 19).

12 234 Fermüller and Aloimonos Figure 18. The normal motion bands corresponding to vectors o length zero intersect on the zero motion curve (curve segment marked by circles). Figure 19. Copoint vectors (r, s). The coaxis vectors are deined with respect to a direction in space. The (A, B, C) coaxis vectors are deined as ollows: A line through the image ormation center deined by the directional cosines (A, B, C) deines a amily o cones with axis (A, B, C) and apex at the origin. The intersection o these cones with the image plane gives rise to conic sections. The normal motion vectors perpendicular to these conic sections are called (A, B, C) coaxis vectors. At every point (x, y) a coaxis vector is parallel to the vector ( A(y ) + Bxy+Cx, Axy B(x )+Cy) (see Fig. 20). As in the case o the iso-normal motion vectors, we choose vectors o a given length and direction and evaluate the regions with positive depth measurements. We Figure 20. (A, B, C) coaxis vectors. consider the (r, s) copoint vectors and the (A, B, C) coaxis vectors o length u n (x, y) (with u n (x, y) a unction in x and y). Where W > 0 the ollowing inequalities Z hold: [y(x 0 r) x(y 0 s) x 0 s + y 0 r] [u n (x, y) x 2 (βs + γ ) y2 (αr + γ ) ( αs + xy + βr ) + y(γs +β) ] +x (γr +α) (αr +βs) >0 (15) [ ( ( Cα u n (x, y) y Aγ ) ( Bγ + x Cβ ) ) ] + Aβ Bα (x 2 + y ) [ ( ) ( y 2 Ax0 Bx0 + C xy + Ay ) 0 ( ) + x 2 By0 + C y (B +Cy 0 ) ] x (A +Cx 0 )+ Ax 0 + By 0 > 0 (16) For u n (x, y) = 0 we obtain regions deined by a line and a conic section. In the case o the copoint vectors the line separates the translational components and the conic separates the rotational components o the vectors. The line passes through the FOE and also through the point (r, s), and thus it can be described by only one unknown. The conic is speciied by only two unknowns, α γ and β γ. In the case o the coaxis vectors,

13 On the Geometry o Visual Correspondence 235 and normal motion ields which are due to rigid motion. These constraints can be exploited in a variety o algorithms dealing with visual navigation tasks. In particular, they could be used to veriy the rigidity o a given motion ield in order to detect and localize independent motion; to estimate the 3D rigid motion or provide bounds or the motion parameters that in conjunction with other algorithms could lead to accurate estimation; and to address problems in visuomotor control in robotic applications such as servoing or stabilization. In the remainder o this section we will describe in more detail ideas that could be used in algorithmic procedures or 3D motion estimation. Figure 21. Separation o (A, B, C) coaxis vectors whose directions are shown in Fig. 20: A line passing through the AOR separates the positive and negative rotational components. A conic through the FOE separates the positive and negative translational components. In the area marked by horizontal lines all (A, B, C) coaxis vectors are greater than zero. In the area marked by vertical lines all (A, B, C) coaxis vectors are smaller than zero. The two other areas contain all the (A, B, C) coaxis vectors o length zero. the line separates the rotational components. It passes through the AOR and through the point (r, s) and thus it is also described by only one unknown. The conic separates the translational components. It is deined by the two coordinates o the FOE, (x 0, y 0 ). The intersection o the lines and the conics lies on the zero motion curve. One o the two other regions deined by the above described curves contains only vectors o length greater than zero, and the other contains only vectors smaller than zero. An illustration is given in Fig. 21, which shows these regions or the class o coaxis vectors displayed on Fig. 20. It becomes clear that the normal motion vectors o same length and direction can be considered as special cases o the copoint vectors. They represent the copoint vectors, or which r and s both are and r s = n y n x.a special class o coaxis vectors, those which correspond to an axis parallel to the XY-plane, is very similar to the iso-normal motion vectors. For all classes o coaxis vectors (A, B, 0) the slope o the line separating the rotational components is A and the conics are all hyperbolas. B 5.1. Using Motion Vectors Knowledge o any single iso-motion contour is suicient to derive the 3D motion. Thus the localization o any such contour provides the motion parameters. Iso-motion contours could be localized with simple matching techniques using ilters which are tuned to respond to image points in dierent rames being a certain distance apart (see Fig. 22). Finding motion vectors o a certain length and direction by means o predeined ilters is a decision problem and thus easier and o lower complexity than computing a motion ield. 5. Applications The constraints developed in this paper allow us to make explicit aspects o the structure o motion ields Figure 22. To test whether points A = (x, y) and B = (x+u, y+v) correspond, we test how well two windows w A and w B placed around the point A and B match. With a set o correspondence operators tuned to dierent values (u,v) the points on the (u,v) iso-motion contours are located.

14 236 Fermüller and Aloimonos In addition, the various relationships between isomotion contours described in Section 3 can be used to increase the accuracy o the computation o iso-motion contours. We can describe every point as the intersection o a certain class o iso-motion contours: the FOE is the intersection o all iso-motion contours; dierent ku iso-motion contours intersect at a point on the zero motion contour; and any point on a general C u iso-motion contour can be considered as the intersection o a amily o iso-motion contours as described in Eq. (7). Furthermore, knowledge about the directions o the motion vectors along iso-motion contours can be o use in the veriication o the contours correct locations. As described in Section 3.2, or every point along any u iso-motion contour the ollowing holds: I we subtract u rom the motion vector v at P, we obtain a vector v, which is parallel to the translational motion component o v. In particular, or every point along the zero motion contour the motion vector is in the direction o the translational motion component. The preceding discussion demonstrates that even i there is no iso-motion contour available, but we have at our disposal a set o points (at least two) with their associated motion vectors, and we know on which iso-motion contour each point lies, the position o the FOE is uniquely determined (as shown in Figs. 6 and 7) Using Normal Motion Vectors Similarly, the iso-normal motion areas as well as the regions containing coaxis and copoint vectors o a certain length and direction can be used in the recovery o the 3D motion. The normal motion regions can be obtained by localizing their boundaries. In particular, the normal motion vectors o length zero are bounded by curves which can be described by only three parameters. Thus a simple search technique in a three-dimensional space, such as in Fer muller (1993), Fer muller and Aloimonos (1993), could be employed to ind the 3D motion parameters. The iso-normal motion areas, however, also allow to obtain bounds on the solutions or the motion parameters in very simple ways. Computations o this kind might be used in combination with other techniques, such as the search techniques described above, or example as preprocessing modules to reduce the amount o search. A possible area or the FOE can be obtained by using iso-normal motion vectors in dierent directions. The normal motion vectors o a given value are located within the normal motion bands (see Fig. 16). The normal motion bands consist o two areas meeting at the point S un. Through S un and the FOE passes the boundary line o the iso-normal motion areas, which also separates the translational motion components (Fig. 11). The slope o the line is known; it is perpendicular to the direction o the normal motion. The exact position o the line is deined by the location where the normal motion band is thinnest. It may not be possible to locate one point and thus the exact line, but only a bounded area which contains the line. Since all isonormal motion areas o dierent lengths but the same direction deine the same line, this bounded area can be located by means o a number o iso-normal motion bands corresponding to parallel motion vectors. The intersection o at least two such bounded areas corresponding to normal motion vectors in dierent directions gives an area in which the FOE lies. This is demonstrated in Fig. 24. Figure 23 shows the synthetic normal low ield, which was used as input data or the localization o the translation and rotation axis demonstrated in the ollowing three igures. (Figs. 24, 25, 26). The image size is , the ocal length is 100 pixel, the origin is at the center o the image. The translation was ( 0.1, 0.1, 0.5) and the rotation was ( , 0.01), where the depth was chosen randomly in an interval between 20.0 and Figure 23. Synthetic normal motion ield used as input data or the localization o the FOE and AOR shown in the next three igures.

15 On the Geometry o Visual Correspondence 237 Figure 24. Localization o FOE: (a) Normal motion band due to normal motion vectors o a certain length parallel to the x-axis and corresponding curves g(x, y) and h(x, y) deining the normal motion area. By localizing, where this normal motion band is thinnest a bounded area or the line separating the translational normal motion components is ound (marked by diagonal lines). (b) Normal motion band due to normal motion vectors o a certain length parallel to the y-axis with overlaid curves g(x, y) and h(x, y) deining the normal motion area and localization o bounded area or the line separating the translational normal motion components (marked by diagonal lines). (c) The intersection o these areas gives a bounded area or the FOE. Instead o ocusing on the thinnest location o the normal motion bands, one could consider various normal motion bands corresponding to dierent lengths but the same direction. Two such iso-normal motion areas cannot intersect in an area that contains the line. Thus an area or the line passing through the FOE can be located as the region between regions where isonormal motion areas intersect (see Fig. 25). Just as a line passing through the FOE bounds the isonormal motion regions, a line passing through the point where the rotation axis pierces the image plane (AOR) bounds regions separating the ( A, B, 0) coaxis vectors o length zero. By inding the locations where the areas separating the ( A, B, 0) coaxis patterns are thinnest and intersecting dierent such areas, a bounded region or the AOR can be located (see Fig. 26). In general one cannot solely rely on single bands o normal motion vectors to locate bounding areas or the FOE and AOR, simply because the structure o the bands also depends on the distribution o the image gradients. I in a certain area there are no gradients available in a desired direction, the appropriate band will not exist there (or be thin). On the other hand, a complete search in subspaces o the rigid motion parameters or the boundaries o the iso-normal motion regions (or the areas with normal low values greater u n and smaller u n, or areas deined by the coaxis vectors) will always lead to correct solutions. The essential inormation about the location o the regions boundaries lies in the location o the iso-normal vectors. Thus a complete search will not be necessary and ideal strategies will be combination algorithms that can make eicient use o the relevant inormation without missing solutions. The ollowing experiments give a demonstration o the localization o bounded areas or the FOE and AOR using two o the algorithms described above or a calibrated motion sequence. Figure 27(a) shows the irst image o the sequence and Fig. 27(b) shows the normal motion ield computed between the irst and the second rame. The image size is , the FOE is at (x, y) = (197, 136) and the AOR is at(x, y) = (413, 124) (with the origin o the coordinate system at the let upper corner). In Fig. 28(a) the points with iso-normal motion vectors parallel to the y-axis o three dierent lengths (0.3, 0.1, and 0.1 pixels) are displayed in red green and blue. These normal motion vectors constrain the FOE to lie in an area bounded by the two horizontal lines where there is no intersection o the dierently colored isomotion areas, in between the regions o intersection. Similarly, the normal motion vectors parallel to the

16 238 Fermüller and Aloimonos Figure 25. Localization o FOE: The FOE cannot lie within the areas where the normal motion bands corresponding to parallel normal motion vectors intersect. (a) Normal motion vectors parallel to the y-axis o three dierent lengths (displayed in dark, medium, and light grey). (b) Polygonal approximation o the boundaries o normal motion bands and localization o an area in which the FOE can lie. x-axis (o length 0.3, 0.0 and 0.3) constrain the FOE to lie in an area bounded by two vertical lines as shown in Fig. 28(b). The intersection o the areas ound in Figs. 28(a) and (b) gives a bounded area or the FOE (see Fig. 28(c)). Figure 29 shows the results o the algorithm using ( A, B, 0) coaxis vectors o length zero to locate the AOR by inding the locations where the bands are thinnest. Figures 29(a) (d) show the results or the the (1, 0, 0) coaxis vectors, (0, 1, 0) coaxis vectors, and

17 On the Geometry o Visual Correspondence 239 locate iso-motion contours normal motion vectors can be employed. For example, the intersection o normal motion bands (belonging to normal motion vectors originating rom one motion vector) gives a segment o the corresponding iso-motion contour (see Figs. 14 and 15). 6. Conclusions The motion ield or the displacement ield due to rigid motion on a system s retina possesses a global structure that is independent o the scene in view and depends only on the parameters o the underlying 3D motion. In this paper we studied this structure by analyzing the geometry o the iso-motion contours and iso-motion regions, i.e., the loci on the retina where the motion vector (or the normal motion vector) could obtain a certain value. We ound that the iso-motion curves and the boundaries o the iso-normal motion areas are o second order and their orm depends on the 3D motion parameters. The theory described here can ind several applications in problems o visual motion interpretation as well as calibration, or in general, in problems related to the matching o two views and its interpretation (Fernmüller and Aloimonos, 1996). Appendix A Iso-Motion Contours and Iso-Normal Motion Area or Spherical Projection Figure 26. Localization o the AOR with (A, B, 0) coaxis vectors: (a) (0.707, 0.707, 0) coaxis vectors o length zero and curves separating the positive rom the negative translational and rotational normal motion components. (b) Localization o a bounded area in which the AOR lies. (0.707, 0.707, 0) coaxis vectors and their intersection in red, green and blue Combined Use o Motion and Normal Motion O course, the iso-motion and iso-normal motion constraints can also be used in a combined manner. To Today s commercially available electronic cameras used in robotic systems and utilized in our experiments have image suraces that are best approximated by a plane. For this reason the analysis in this paper was perormed or a planar retina as well. However, rom a geometric point o view, or the purpose o analyzing visual motion globally, the use o a spherical retina is much more natural. This is because it provides us with a 360 ield o view. On the sphere the iso-motion contours and the iso-normal motion areas take a simple orm. Their derivation rom irst principles provides valuable geometric intuition into the problem and is described here. On a sphere o radius (see Fig. 30), the image r o every point R is r = R R with R being the norm o the vector R; Recalling rom Eq. (1), i we project the 3D-motion on the sphere the

18 240 Fermüller and Aloimonos Figure 27. (a) First rame o a motion sequence. (b) Normal motion ield. motion vector r can be expressed as r = 1 ( ) 1 (t r)r t ω r. (A1) R When employing general imaging suraces, the motion vectors at a surace point are deined on the local tangent plane. In the discussion o u iso-motion contours on the plane, we used a simpliied notation and did not mention that the vector u actually results rom the projection o a 3D vector on the local tangent plane which happens to be the same everywhere. However, or the case o a spherical retina we have to be more general and provide deinitions in the orm o projections o 3D vectors on the local tangent plane. In the ollowing we use the negative value o the projection o a vector. Thus, or example, i u is a 3D vector, the u iso-motion contours on a sphere with radius one ( = 1), are deined as the locus o points r on the sphere where the motion vector has the value (u r)r u. The ollowing our propositions characterize the structure and orm o the iso-motion contours (Propositions 1 3) and iso-normal motion areas (Proposition 4) on the unit sphere: Proposition 1. For a rigid motion ield on the sphere the geometrical locus o points at which the motion vector could have the value (u r)r u is described by a second order equation that in general gives rise to one or two curves. These curve(s) pass through the FOE and the FOC (the points where t pierces the sphere) and the points R u (R u1 and R u2 )(where the rotational component o the motion is equal to (u r)r u). P roo: For the locus o points r with motion vector (u r)r u using (A1) we obtain 1 ((t r)r t) (ω r) = (u r)r u. R (A2) I u = 0 we obtain ((t r)r t) (ω r) = 0 or (t r)(r ω)r + (t ω)r = 0 and thus t ω (t r)(r ω) = 0. (A3) Otherwise, projecting both sides o (A2) on the vector ((ω r)r ω), provided that ω r 0 and on the vector u r, provided that u r 0 gives: and 1 (t ω (t r)(r ω)) R = u ω (u r)(ω r) (A4) 1 t (u r) = u ω (u r)(ω r) (A5) R Equating the let-hand side o Eq. (A4) with the lethand side o Eq. (A5) we derive (r ω)(r t) (t ω) + (u t) r = 0 (A6) Each o the Eqs. (A3) and (A6) describes a second order curve on the sphere and gives the locus o points, where the motion vector could take the value (u r)r u.