The Appearance of Surfaces Specified by Motion Parallax and Binocular Disparity

Transcription

1 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 1989,41A (4) The Appearance of Surfaces Specified by Motion Parallax and Binocular Disparity Brian J. Rogers University of Oxford Thomas S. Collett University of Sussex The experiments reported in this paper were designed to investigate how depth information from binocular disparity and motion parallax cues is integrated in the human visual system. Observers viewed simulated 3-D corrugated surfaces that translated to and fro across their line of sight. The depth of the corrugations was specified by either motion parallax, or binocular disparities, or some combination of the two. The amount of perceived depth in the corrugations was measured using a matching technique. A monocularly viewed surface specified by parallax alone was seen as a rigid, corrugated surface translating along a fronto-parallel path. The perceived depth of the corrugations increased monotonically with the amount of parallax motion, just as if observers were viewing an equivalent real surface that produced the same parallax transformation. With binocular viewing and zero disparities between the images seen by the two eyes, the perceived depth was only about half of that predicted by the monocular cue. In addition, this binocularly viewed surface appeared to rotate about a vertical axis as it translated to and fro. With other combinations of motion parallax and binocular disparity, parallax only affected the perceived depth when the disparity gradients of the corrugations were shallow. The discrepancy between the parallax and disparity signals was typically resolved by an apparent rotation of the surface as it translated to and fro. The results are consistent with the idea that the visual system attempts to minimize the discrepancies between (1) the depth signalled by disparity and that required by a particular interpretation of the parallax transformation and (2) the amount of rotation required by that interpretation and the amount of rotation signalled by other cues in the display. Requests for reprints should be sent to Brian J. Rogers, Department of Experimental Psychology, South Parks Road, Oxford OX1 3UD. Dr. Vicki Bruce acted as editor for this paper. ~ ~ The Experimental Psychology Society

2 698 ROGERS AND COLLETT The human visual system is able to exploit a variety of different sources of information for the perception of three-dimensional surfaces (Sekuler & Blake, 1985; Bruce & Green, 1985). These sources include binocular stereopsis and motion parallax, as well as the so-called pictorial cues of shading, perspective and occlusion. A common objective of many previous studies of 3-D perception has been to determine the effectiveness of the different cues when presented in isolation. More recently, however, attention has also been focussed on the problem of how 3-D information from different sources is combined when interpreting a visual scene (Buckley, Frisby, & Mayhew, 1988; Harris, 1980; Marr, 1982; Richards, 1985). In the present paper, we have approached this question by investigating the appearance of horizontally corrugated surfaces in which the 3-D structure was specified by various proportions of motion parallax and binocular disparity. Individually, both motion parallax and binocular disparity can produce vivid and accurate impressions of the 3-D structure of objects and surfaces (Braunstein, 1968; Helmholtz, 1924; Julesz, 1971; Rogers & Graham, 1979; Wallach & O Connell, 1953; Wheatstone, 1838). In addition, Rogers and Graham (1982, 1983, 1985) have found striking similarities in the perception of surfaces specified by these two cues. For example, the shapes of the sensitivity functions that show depth modulation thresholds as a function of the spatial frequency of the 3-D corrugations are very similar for motion parallax- and disparity-specified surfaces (Rogers & Graham, 1982, 1985). Furthermore, Graham and Rogers have demonstrated that there can be interactions between the two cues in that binocular disparities can be used to cancel the depth aftereffects produced by prolonged viewing of monocular parallax surfaces and vice versa (Graham & Rogers, 1982). In spite of these similarities, the cues of disparity and motion parallax differ in one important respect. The interpretation of a given pattern of image motion is always ambiguous in a way that the interpretation of binocular disparity is not. This ambiguity arises because the interpretation of structure from motion is based on changes in the retinal image that occur over time, and there is always the possibility that changes in structure and form may also have taken place. Hence, any given pattern of motion on the retina could have been produced by an infinite number of possible situations in a 3-D world, including the special case of a 2-D array of moving points, such as that generated on a flat television screen (Adelson, 1985). Binocular disparities, on the other hand, are present in the differences between simultaneous images on the two retinae and so do not share this spatio-temporal uncertainty. Despite this ambiguity, which is a feature of all perspective transformations, there is typically little or no ambiguity in what we perceive. Those patterns of relative motion that simulate a rigid 3-D surface seen from slightly different directions (observer-produced parallax) or a 3-0 surface

3 MOTION PARALLAX AND BINOCULAR DISPARITY 699 translating across the observer s line of sight (object-produced parallax) are usually seen as precisely that (Braunstein, 1966, 1968; Rogers & Graham, 1979). The fact that only a single (and usually rigid) interpretation of a particular pattern of image motion is entertained by the visual system has been cited as evidence for the operation of a rigidity principle or constraint in the interpretation of perspective transformations (Johansson, 1973; Ullman, 1979). The primary objective of the experiments described below was to measure the magnitude of perceived depth in 3-D corrugations specified by different amounts of motion parallax and binocular disparity and to determine how the visual systems copes with the apparently contradictory information about the structure of 3-D surfaces (Rogers & Collett, 1985). Methods In all the experiments described below, observers viewed a horizontally corrugated test surface that translated to and fro across the observer s line of sight. The 3-D corrugations were specified by either (1) relative motion alone, or (2) binocular disparity alone, or (3) the two together in varying proportions. The observers compared the appearance of a particular test surface with a standard matching surface that always contained equal amounts of disparity and relative motion (as would be generated by a real 3-D surface translating across the line of sight). The observer s task was to adjust the peak-to-trough amplitude of the corrugations in the standard matching surface (with congruent disparity and parallax) until the test and matching surfaces were judged to have the same amplitude. Display. The 3-D surfaces were generated using a pair of large screen display oscilloscopes (20 x 25 deg) viewed independently by the two eyes in a modified Wheatstone stereoscope configuration (Figure 1). Identical 256 x 256 pixel arrays of random dots were displayed on the two screens. When fused, the dots were seen as a single flat array in the plane of the screen. Horizontal disparities were introduced by applying additional sinusoidal signals, 180 out of phase, to the X-inputs of the two oscilloscopes (Rogers & Graham, 1982). The peak-to-trough depth of the disparity corrugations was controlled by the amplitude of the additional X-signals. The motion parallax information simulated that which would be generated by a 3-D corrugated surface translating along a linear path across the observer s line of sight (object-produced parallax) (Rogers & Graham, 1979). To achieve this, the display oscilloscopes were placed on a platform suspended from the ceiling, which was swung from side to side along a fronto-parallel path. The random dot arrays were distorted in synchrony with the oscilloscope movements to produce the appropriate parallax

4 700 ROGERS AND COLLETT FIG. 1. Perspective sketch of the apparatus. Front-silvered mirrors allowed each eye a view of only one oscilloscope screen. The oscilloscopes were swung together on a platform along a fronto-parallel plane as indicated by the arrows. transformations. The peak-to-peak amplitude of the oscilloscope movements was 13 cm (1 3" visual angle) and the period of oscillation was about 3 sec. The parallax transformations of the random dot arrays were generated by applying identical additional sinusoidal signals to the X-inputs of the two oscilloscopes and modulating the amplitude of the signals according to the horizontal position of the oscilloscopes (Rogers & Graham, 1979, 1982). The consequence of this was to produce relative motion between different rows of dots, which mimicked the parallax that would be produced by a real corrugated surface in the plane of the screen and moving with it. The peakto-trough depth of the parallax corrugations was controlled by the gain of the amplitude modulation. To make it easy to compare the relative amounts of disparity and parallax present on a given trial, the parallax amplitude is expressed as an equivalent binocular disparity. This was achieved by integrating the amount of relative motion produced when the oscilloscopes translated through the convergence angle of the eyes at this distance (6.5") (Rogers & Graham, 1982). To study the perceived depth of monocular corrugations specified by parallax alone, one or other of the display screens was blanked electronically. In all other conditions, the random dot patterns were visible on both screens. In most experiments, the spatial frequency of the corrugations, in both test and matching surfaces, was 0.2 cycles per deg. The experiments were all carried out under subdued lighting conditions, which

5 MOTION PARALLAX AND BINOCULAR DISPARITY 701 enabled the observers to see the frame of the oscilloscope screens and the surrounding surfaces. Procedure, During each experiment, the observer rested his/her head on a chin rest and was required to track the translating surface as it oscillated to and fro across the line of sight. While watching the matching surface, the observer turned a potentiometer to adjust the peak-to-trough depth of the corrugations until they appeared to have the same amplitude as those of the test surface. The test surface was displayed on the oscilloscope screens for 3 sec and was then replaced by the matching surface for a further 3 sec. This cycle continued until the observer was satisfied with the match. There were three observers, including the two authors. EXPERIMENT 1 SURFACES SPECIFIED BY MONOCULAR MOTION PARALLAX The first experiment was designed to measure the perceived depth and apparent path of a corrugated test surface specified by monocular motion parallax alone. Simple geometrical considerations show that when a real corrugated surface translates rigidly along a fronto-parallel path, the amount of motion parallax is a monotonic function of the peak-to-trough depth of the corrugations (the function is linear for small amplitudes). Hence, if the visual system makes the assumption that the path of the translating surface is linear and fronto-parallel (or has additional information that it is so), then the amount of depth in the corrugations can be predicted from the amount of parallax motion. However, if the path of the translating surface is unspecified, then the apparent depth of the corrugations will be underconstrained, with many different values possible, as we explain below. Thus our measurements of apparent depth can be used to gauge whether the visual system interprets a given combination of parallax and disparity as a rigid surface moving linearly in a fronto-parallel plane. Results The observers in the first experiment viewed the test surface monocularly and adjusted the amplitude of the binocularly viewed matching surface until the corrugations of the two surfaces appeared to have the same peak-to-trough depth. The perceived depths of the corrugations for different amounts of relative motion plotted in Figure 2 (closed symbols) are the averages of the settings of the three subjects. The variability of the subject means was sufficiently small and the overall pattern of results sufficiently similar, in both this and subsequent experiments, to justify averaging the results over

6 702 ROGERS AND COLLETT subjects. Both relative motion and perceived depth are expressed in terms of equivalent disparities. The striking feature of these results is the closeness of fit between the perceived amplitude of the corrugations of the test surface and the predicted amplitude based on the amount of relative motion, together with the assumption of a rigid fronto-parallel path of translation (dashed line). The slope of the best-fitting regression line is 0.93, and the Pearson correlation coefficient is This finding demonstrates how precisely the visual system interprets the parallax information as a surface that is rigid and moving linearly in a fronto-parallel plane. Not surprisingly, this is what observers report. The perceived surface did not appear either to deform or to rotate as it translated to and fro. EXPERIMENT 2 SURFACES SPECIFIED BY MOTION PARALLAX AND D IS PAR ITY 1. Binocular Motion Parallax with Zero Disparity As a first step in studying the interactions between parallax and disparity cues, observers were presented with a surface that was specified as corrugated according to the parallax information but flat according to binocular disparities. Identical parallax transformations were generated on both oscilloscope screens as they translated across the observer s line of sight, but because the random dot patterns on the two oscilloscopes were identical the disparities between corresponding elements of the two patterns were always zero. The observer s task was again to set the perceived depth of the matching surface (with equal amounts of parallax and disparity) so that its corrugations had the same apparent amplitude as those of the binocularly viewed test surface. Results The results are shown in Figure 2 (open symbols). The perceived depth of the corrugations increased monotonically with increasing amounts of parallax, but the perceived amplitude of the corrugations was only about 50% of that found when the same surface was viewed monocularly (closed symbols). One interpretation of this result is that the perceived depth of the corrugations was roughly the average of (1) the depth specified by motion parallax (assuming rigidity and a fronto-parallel path), and (2) the zero depth specified by the binocular disparities. A second feature of the appearance of the test surface in this condition is particularly important. The perceived corrugations were seen to rotate about

7 MOTION PARALLAX AND BINOCULAR DISPARITY 703 Matched amplitude -b min arc n Parallax amplitude -min arc -10 0, -14 FIG. 2. The apparent peak-to-trough depth of corrugations (ordinate) as a function of the amount of motion parallax (abscissa), measured using a matching procedure. Closed symbols (0) indicate results for monocular parallax condition with one oscilloscope screen blanked. Open symbols (0) indicate results for binocular parallax condition with zero disparities between corresponding elements on the two screens. Each data point represents the mean of a total of six readings from each of three subjects. a vertical axis through the centre of the screen as they translated across the observer s line of sight. In this case, the surface appeared to rotate in a clockwise direction as it translated from the right to the left in what we shall call a convex direction with respect to the observer, as illustrated in Figure 3d. Why should the surface appear to rotate in this manner? Consider the parallax transformation generated by a real 3-D corrugated surface translating along a fronto-parallel path (Figure 3a and b). To a first approximation, the same pattern of relative motion would also be generated by a surface with more (or less) peak-to-trough depth that simultaneously rotated about a vertical axis as it translated to and fro. More specifically, a similar pattern of relative motion would be produced by a corrugated surface with more depth, which rotated in a concave direction with respect to the observer (Figure 3c), or a surface with less depth, which rotated in a convex direction with respect to the observer (Figure 3d). The use of convex and concave is meant to convey the direction of rotation that accompanied the translation

8 704 ROGERS AND COLLETT b / C concave / d convex / e FIG. 3. Alternative ways of interpreting the same parallax transformation. (a) shows the amount of relative motion between the centre band of dots and the surround. This amount of relative motion would be produced by: a rigid surface translating along a fronto-parallel plane with a given amount of peak-to-trough depth (b); or a corrugated surface with more depth, which simultaneously rotates during translation in a concuve direction (c); or a corrugated surface with less depth, which rotates in the opposite or convex direction during translation (d); or a surface with corrugations 180 out of phase, which rotates in a markedly concave direction with the centre of rotation in front of the observer (e).

9 MOTION PARALLAX AND BINOCULAR DISPARITY 705 of the surface and does not imply any deviation of the path away from the fronto-parallel. Hence, in the binocular parallax situation described here, the combination of a smaller amplitude of the perceived surface (50%), together with an apparent convex rotation, is still consistent (to a first approximation) with the amount of parallax motion in the display. Moreover, the reduced amplitude of the perceived depth suggests that the visual system takes some notice of the absence of depth specified by the disparity cues. This explanation of how it is possible to have several interpretations for a given parallax transformation is couched in terms of the peak-to-trough amplitude of the corrugations. The same conclusion can also be reached by considering changes in the local features of the retinal image (Koenderink, 1985; Rogers & Koenderink, 1986). In particular, consider the changes in the retinal orientation of a local line element on a surface that translates along a fronto-parallel path. If the surface translates through f 6.5 cm at a viewing distance of 57 cm, the retinal orientation of the element will change by f a x sin 6.5, where a is the angle of slant of the surface with respect to the vertical (Figure 4a). However, if the surface simultaneously rotates in a concave direction by f 6.5 (such that it is always facing the observer), there will be no orientation change of a local line element, whatever the slant of the surface. In general, the change of orientation of a local line element will be f b x sin 0, where 0 is the angle of rotation of the surface about a vertical axis as seen by the observer and b its slant (Figure 4). It holds, therefore, that equivalent changes in the orientation of retinal elements will be produced whenever the following relationship is true: a x sin 6.5 = b x sin 0 (1) where a is the angle of slant of a fronto-parallel translating surface; b is the angle of slant of an equivalent surface, and 0 its degree of rotation about a vertical axis with respect to the observer. Hence the predicted angle of rotation of the equivalent surface, 6, will vary with the ratio a/b. a. sin 0 = -sin 6.5 b Some of our experimental observations, reported later, suggest that an analysis in terms of surface slant or depth gradient is a better predictor of the results obtained in situations where the amounts of parallax and disparity are unequal than an analysis based on peak-to-trough amplitude. Occasionally during this experiment the phase of the corrugations reversed, so that the peaks became troughs and the troughs turned into peaks. When this happened, the peak-to-trough amplitude of the corrugations looked roughly similar in magnitude, but the surface now appeared to

10 ROGERS AND COLLETT 4- a ,. :. C.:: _.. J J 0...,....,......,I......,. FIG. 4. The same change in the retinal orientation of a local line element lying on a surface would be produced by a surface slanting at an angle a to the vertical, which translates along a fronto-parallel path (a); or a surface slanting at a larger angle, which simultaneously rotates during translation, in a concave direction (b), or a surface slanting at a smaller angle, which simultaneously rotates in the opposite or convex direction during translation (c). rotate in a concave rather than convex direction. Figure 3e shows that, again to a first approximation, there is a possible interpretation of the parallax motion in which the opposite phase of the corrugations is seen together with a concave rotation of the surface as it translates to and fro across the observer s line of sight. 2. In-phase Parallax and Disparity The results of the previous experiment suggest that there is a compromise in the perceived depth between that indicated by the parallax and disparity cues. This finding raises the question of whether the compromise holds for surfaces specified by non-zero disparities. To answer this question, observers performed a similar matching task for some 30 different test surfaces in

11 MOTION PARALLAX AND BINOCULAR DISPARITY 707 which the corrugations were specified by different combinations of motion parallax and binocular disparity. As in the previous experiments, the variable amplitude matching surface always contained equal amounts of parallax and disparity and appeared to translate along a fronto-parallel path without rotation. Results The results of this experiment are shown in Figure 5, where the matched depth of a particular combination of parallax and disparity is plotted against the amplitude of the parallax component (abscissa). The separate curves represent different amplitudes of the disparity component. With zero disparities between the two displays of the test surface (filled circles), the apparent depth of the test surface was roughly half of that specified by the amplitude of the parallax component (assuming a rigid, fronto-parallel path), as has already been described. However, when the test surface contained non-zero peak-to-trough disparity in addition to the parallax motion, the apparent amplitude of the corrugations only increased slightly with increasing parallax amplitude. This is indicated in Figure 5 by the small positive slopes of the functions that relate the apparent depth to the parallax amplitude. In addition, the slope of these curves becomes smaller as the disparity Matched amplitude min arc.- lo[, "' FIG. 5. The apparent peak-to-trough depth of binocularly viewed corrugations specified by different amounts of motion parallax (abscissa) and disparity (parameter of different curves). The upper right quadrant presents results for conditions in which the phase of the corrugations specified by the parallax and disparity cues was the same. The upper left quadrant presents results for conditions in which the depth indicated by the parallax and disparity cues was our of phase. Each data point represents the mean of a total of six readings for each of three subjects.

12 708 ROGERS AND COLLE IT component increases. Overall, the data plotted in the upper right quadrant of Figure 5 are crudely fitted by the expression: D=S+ P 2x(l+s) (3) where D = perceived depth. S= stereo (disparity) amplitude, and P= parallax amplitude. This equation merely redescribes the experimental findings that (1) there is an approximately linear relationship between matched depth and parallax amplitude and (2) the intercept and gradient of the line depend on the amplitude of the disparities present. It is descriptive rather than based on any particular theoretical model. Consequently, one way of describing the results would be to say that the principal determinant of perceived depth is the magnitude of the disparities present and that the amount of parallax has only a small effect on perceived depth, particularly when the peak-to-trough disparities are large. This is not to suggest that the motion parallax information is either ignored or suppressed. As with the zero disparity surfaces of the previous experiment, subjects reported that the corrugated surface, with a particular amount of depth, appeared to rotate about a vertical axis through the centre of the pattern as it translated to and fro. The combination of perceived depth and perceived rotation appeared to remain qualitatively, and possibly quantitatively, consistent with the magnitude of parallax present. For example, for surfaces specified by binocular disparities but with no relative motion between the peaks and troughs (points falling on the ordinate of the graph), the corrugations appeared to remain orrhogonal to the line of sight (i.e. always facing the observer) as they translated to and fro. The only situation in which a real corrugated surface could create binocular disparities, withour generating any motion parallax between the peaks and troughs of the corrugations, would be one in which the surface simultaneously rotated in a concave direction, always orthogonal to the observer s line of sight. Moreover. one would expect the corrugations of such a surface to have much the same apparent depth as those of a surface with the same amount of disparity together with an equivalent amount of motion parallax. And this was indeed the case (Figure 5). These results show that the functions that relate the apparent depth of the corrugations to the amount of motion parallax become flatter as the size of the disparity component increases (Figure 5). Does this change in slope depend on the peak-to-trough amplitude of the disparity signal, or the maximum disparity gradient present, both of which increase with the size of the disparity signal? To dissociate amplitude and gradient, we repeated the matching experiment using corrugations of three different spatial frequencies (0.1, 0.2, and 0.4 cycles/deg). The peak-to-trough amplitude of the disparity

13 MOTION PARALLAX AND BINOCULAR DISPARITY 709 component was fixed at 2min arc, and the amplitude of the parallax component varied from 0 to 12 min arc. If the relation between apparent depth and motion parallax depends only on the peak-to-trough amplitude of the disparity signal, the slopes of the functions should be invariant with spatial frequency. On the other hand, if disparity gradient is the critical variable, the slopes of the functions should become steeper as spatial frequency (and the disparity gradient) decreases. The results in Figure 6 show that the slope does indeed become steeper with decreasing spatial frequency. The relation between perceived depth and parallax is steepest for the corrugations with the smallest gradient tested (0.1 cycles/deg) and shallowest for the corrugations with the largest gradient (0.4 cycles/deg). Thus the gradient of disparity appears to be the more important factor in determining how much depth is induced by a given parallax component. Moreover, the 0.1 cycles/deg surface, which produced the most significant increase in perceived depth with increasing parallax, was also perceived as undergoing the largest amount of convex rotation as it translated to and fro, and the 0.4 cycles/deg surface produced the smallest amount of apparent rotation. 3. Out-of-phase Parallax and Disparity The data points in the upper left quadrant of Figure 5 represent those conditions in which the gradients of the parallax and disparity signals were of opposite sign. Thus when the disparities indicated a peak, the parallax motions (assuming a rigid, fronto-parallel path) indicated a trough, and viceversa. Under these circumstances, the perceived phase of the corrugations lo 1 Matched amplitude min arc + I,,,,,,, Parallax amplitude min arc FIG. 6. The apparent peak-to-trough depth of binocularly viewed corrugations as a function of the amount of parallax (abscissa) and the spatial frequency of the corrugations (parameter of different curves). Peak-to-trough disparity was held constant at 2 min arc.

14 71 0 ROGERS AND COLLETT was determined by the direction of the disparity gradients, so that convergent disparities were seen as peaks and divergent disparities as troughs. Motion parallax, on the other hand, determined the direction and amount by which the surface was seen to rotate. Under these conditions, the reported appearance was of a surface undergoing a concave rotation with respect to the observer as it translated to and fro. Again, this percept corresponds to the only possible real surface that could produce parallax motions and disparities of opposite phase (Figure 3e). To take these results into account, Equation (3) must be modified so that: D=S+ IPI 2x(I+s) (4) There is, in fact, a small departure from the symmetry of the curves in the two quadrants of Figure 5 that is not predicted by this formulation and for which we have no adequate explanation. The results show that even the smallest disparity modulation we tested (2 min arc peak-to-trough) was capable of specifying the phase of the perceived corrugations. This means that if one starts out with a binocular surface over which there is zero disparity modulation and with the corrugations specified solely by relative motion, then the predominant percept is of in-phase parallax corrugations. The rows of dots moving in the same direction as the oscilloscope are seen as the peaks of corrugations that rotate in a convex direction. However, the addition of just 2 min arc of disparity in anti-phase reverses the apparent phase of corrugations, so that rows of dots moving in the same direction as the oscilloscope movement are seen as the troughs of corrugations rotating in a concave direction. As an additional experiment, we determined the point at which this reversal occurred. The amplitude of the out-of-phase disparity signal was slowly increased from zero, and the observer s task was to press a key when the corrugations flipped from one phase to the other. Phase reversals typically occurred when the peak-to-trough amplitude of the disparity signal reached about 0.8 min arc. This value did not vary significantly with the rate at which the disparity signal increased. Although no formal measurements were taken, the perceived amplitude of the corrugations appeared to be roughly similar before and after the reversal. However, the perceived sense of rotation of the translating surface switched abruptly from convex to concave, as our interpretation of the data would predict. 4. Parallax and Disparity Gradients of Different Spatial Frequencies Finally, we examined the appearance of corrugated surfaces in which the corrugation frequency specified by one of the depth cues was three times that

15 MOTION PARALLAX AND BINOCULAR DISPARITY 71 1 specified by the other. If parallax and disparity cues are truly additive in their perceptual effects, such a mixture of different spatial frequencies should lead to the perception of a surface with corrugations that look more square or more triangular, depending on the phase relationship of the fundamental and third harmonic (Graham & Nachmias, 1971). Two conditions were tested. In the first, the fundamental corrugation frequency (0.2 cycles/deg) was specified by binocular disparities, and the third harmonic (0.6 cycles/deg with a smaller amplitude than the fundamental) was specified by motion parallax. Results With the two sources of information in peaks subtract or sine phase, the presence of the parallax-specified third harmonic made the overall surface look slightly more squared. When the two components were in peaks add or cosine phase, the surface looked slightly more triangular. Most observers could readily discriminate the two phase conditions. However, in addition to producing a change in the perceived shape, the parallax component also manifested itself as apparent relative motion between rows of dots. When motion parallax specified the fundamental component and binocular disparities the third harmonic, the resultant percept depended crucially on the amplitude of the disparity signal. The effect of adding small amplitude disparity modulations of less than 1 min arc peak-to-trough was to superimpose third harmonic ripples onto the low-frequency corrugations. When the amplitude of the disparity component exceeded this value, however, the third harmonic disparity corrugations dominated the appearance of the surface, whatever the amplitude of the parallax signal. The parallax motions were then seen as relative motion between rows of dots rather than as depth. In other words, the combined effects of the two cues were only additive over a limited range of conditions. Again we see that the gradient of disparity may determine how the disparity information is interpreted. When the disparity component provided the third harmonic information (with steeper disparity gradients), it dominated the perceptual outcome at a smaller amplitude than it did when the disparity component provided the fundamental frequency information. DISCUSSION 1. Rigidity and Monocular Motion Parallax Johansson (1 973) and Ullman (1 979) have both argued that the visual system has a bias towards interpreting image motion on the retina in terms of the movement of rigid objects, or an observer moving within a rigid environment. On the face of it, the observations of monocularly viewed surfaces

16 71 2 ROGERS AND COLLETT reported here (in which the 3-D information was provided by motion parallax alone) provide yet another example of the rigidity constraint at work. Subjects saw a rigid corrugated surface moving along a frontoparallel path, and none of the possible non-rigid interpretations was ever entertained. The visual system, however, does not only have the problem of choosing between rigid and non-rigid interpretations. As we discussed earlier, there is not just one, but a whole set of possible rigid interpretations of a given parallax transformation. Each interpretation has a particular amount of peak-to-trough depth and an associated amount of rotation about a vertical axis (Figures 3 and 4). In practice, observers saw only one interpretation from this set with monocular motion parallax: that in which the surface translated linearly along a fronto-parallel path without any rotation out of the fronto-parallel plane. The apparent depth of the corrugations could be predicted precisely from the amount of parallax present and an assumption of no rotation. Why should this particular rigid interpretation be favoured over the other possibilities? We suggest it is because this percept is the only one to match the perspective transformation in all respects. Although the interpretations that include rotation fit the pattern of image motion to a first approximation, they do not do so perfectly. If a real surface rotated about a vertical axis, the horizontal extent of the surface (in angular terms) would change, as would the projected shapes of local and global features that would be subject to a trapezoidal transformation (Koenderink, 1985, 1986; Rogers et al., 1988). As the oscilloscopes in our display actually translated along a fronto-parallel path (rather than a circular path around the observer), the plane of the display screen rotated with respect to the observer s line of sight through f6.5. As a consequence, there are horizontal size and feature changes in the retinal image that are appropriate to, and therefore specify, a fronto-parallel path. Thus, the only interpretation of the pattern of retinal image motion that is consistent with all aspects of the optic flow is the one in which the corrugated surface moves along a frontoparallel path. We suggest, therefore, that under these circumstances the chosen interpretation is forced by the characteristics of the optic flow and not by any perceptual bias towards rigidity or linear motion. 2. Discrepant Parallax and Disparity Signals The interpretation of monocular motion parallax surfaces described above was such that it was fully consistent with all the available information in the display. For surfaces specified by unequal amounts of parallax and disparity, however, there is no totally consistent interpretation. Hence the appearance of surfaces in these circumstances can tell us something of the compromises made by the visual system when confronted with contradictory information.

17 MOTION PARALLAX AND BINOCULAR DISPARITY 71 3 According to the argument presented earlier and depicted in Figures 3 and 4, a particular amount of motion parallax can be interpreted variously as corrugations of greater or lesser amplitude (or gradient), with corresponding amounts of rotation of the surface out of the fronto-parallel plane. If this were the whole story, we might expect that the amount of disparity present would determine the perceived depth, and that the direction and amount of rotation seen would conform to the ratio of the parallax and disparity gradients as indicated by Equation (2). If this were the case, the amount of perceived depth would remain constant with variations in the amount and sign of the parallax information when the data are plotted as in Figure 5. However, our experimental results show that there are small, but reliable, increases in the perceived peak-to-trough depth with increasing amounts of parallax, particularly when the disparity gradients are small. (The slopes of the best-fitting regression lines for the 2 and 4 min arc disparity data, 0.18 and 0.12, are different from 0 at the 1% and 2% significance levels, respectively. The slope of the best-fitting line for the 8 min arc data is not significantly different from 0 at the 5% level). One possible account of these results is suggested by Expression (3), which provides a good fit to the data points. This expression suggests that the amount of perceived depth is calculated by adding a small weighted proportion of the parallax amplitude to the amount of disparity present. The size of the weighting factor, according to this interpretation, decreases as the gradient of the disparity signal increases. For steep disparity gradients, little account would be taken of the parallax signal, whereas for shallow gradients more weight would be given, perhaps because shallow disparity gradient signals are unreliable. According to this scheme, the perceived depth of the corrugations is a joint function of the disparity and parallax signals. The expression itself says nothing about how much and in which direction the surface should appear to rotate, but we might expect that this would depend on the difference between the size of the parallax signal and the predicted amount of depth derived from Expression (3). This account is incomplete and therefore unsatisfactory. It tells us that the visual system relies more heavily on disparity information when disparity gradients are steep than when they are shallow, but it fails to provide any understanding of why a given parallax component should be interpreted in the way it is. To approach this problem, we return to the theme developed in the previous section in which we pointed out that any interpretation of a parallax signal that involves the surface rotating out of the fronto-parallel plane will necessarily be inconsistent with the variety of signals that indicate that the surface is moving along a fronto-parallel path. The visual system is thus faced with a dilemma whenever the amounts of motion parallax and binocular disparity are unequal. If the computed surface slant or gradient is made equivalent to that specified by the parallax cues (assuming a fronto-parallel translation path), there must inevitably be a

18 71 4 ROGERS AND COLLETT discrepancy between the slant specified by parallax and that indicated by the disparities. On the other hand, if the computed surface slant is made equivalent to that specified by the disparities present, there will be an appropriate interpretation of the parallax transformation (Figure 4), but only at the expense of a discrepancy between the amount of predicted rotation and that specified by other characteristics of the optic flow. We propose that the visual system chooses an interpretation of the parallax and disparity signals that minimizes the two kinds of discrepancy simultaneously, viz: (1) the difference between the surface slant predicted by a particular interpretation of the parallax signal and the surface slant indicated by the disparity cues and (2) the predicted rotation required by that interpretation and the amount (or lack) of rotation indicated by other cues. Predictions of the Model There are several signs that this prescription is followed. Take, for example, the case in which the motion parallax information is presented binocularly with zero disparities between all elements of the display. One possible interpretation of the parallax transformation is of a surface with very shallow corrugations, which is more in keeping with the zero disparities (Figure 3d). But this particular interpretation also demands a large rotation of the surface, which would be inconsistent with the lack of rotation signalled by other cues. On the other hand, an interpretation that involves no rotation of the surface out of the fronto-parallel plane is necessarily accompanied by significant depth gradients, which would be inconsistent with the zero disparities present. We suggest that the visual system tries to minimize both these discrepancies simultaneously. A simultaneous minimization of this sort would lead to a compromise in which the surface appears both to rotate and to have a compromise gradient of the corrugations. This was exactly the pattern of results described earlier. The model also predicts that both the amount of depth in the corrugations and the degree of rotation will increase monotonically with the amount of parallax present, as was found experimentally (Figure 2). This explanation of how the visual system copes with parallax and disparity signals of differing amplitude also predicts that the slope of the curve relating perceived depth to the size of the parallax signal should be shallower as the disparity gradients increase (Figure 5). The predicted amount of rotation of a corrugated surface specified by unequal amounts of parallax and disparity depends on the ratio of the parallax and disparity gradients (Equation 2). Hence a small absolute discrepancy in the amounts of parallax and disparity (say, 2min arc) would generate a much larger rotational discrepancy when the disparity gradients are shallow. With steep disparity gradients, the amount of rotation needed to accommodate a given

19 MOTION PARALLAX AND BINOCULAR DISPARITY 71 5 difference in the parallax and disparity gradients is small. Hence both the predicted increase in depth (with an increasing amount of parallax) and the amount of perceived rotation should be small. With shallow disparity gradients, on the other hand, the amount of rotation needed to accommodate the same parallaxdisparity difference is much larger. Hence, if the visual system minimizes both discrepancies simultaneously (by choosing the minimum sum of square errors, for example), there should be both a significant increase in the perceived depth with increasing parallax and the impression of a substantial rotation of the surface, as we reported earlier. A similar argument applies to the appearance of surfaces specified by outof-phase parallax and disparity cues. If the perceived phase and amplitude of the corrugations were driven by the disparity signal alone, the computed surface would have to rotate substantially in a concave direction in order to remain consistent with the parallax signal (Figure 3e). Such a rotation would, however, be inconsistent with the other features of the optic flow that indicate no rotation. This rotational discrepancy, however, could be reduced by making the corrugations deeper than they should be according to the disparity cues alone. The experimental results again suggest that the visual system adopts such a compromise. Subjects perceived the amplitude of the corrugations to be greater than that specified by the disparity signals (upper left quadrant of Figure 5), and the surface was perceived to translate along a concave path. Note that this increase in the amount of perceived depth is in the opposite direction to that predicted by any model that calculated a weighted average of the parallax and disparity amplitudes. Conclusions In our proposed scheme of how the visual system copes with apparently contradictory parallax and disparity information, the predicted depth of the corrugated surfaces is based upon an interpretation of the parallax information that minimizes both the surface gradient and rotational discrepancies. This hypothesis predicts the differing slopes of the functions in Figure 5, as well as the smaller than expected increase in perceived depth under binocular parallax conditions (Figure 2). However, because we were only able to measure the amount of perceived rotation qualitatively, we cannot be sure that the amount of perceived rotation corresponds to that predicted by a given minimization procedure, in any particular case. Our simulation also assumes that all the parallax information is used in the trade-off between rotation and depth. There are obviously other minimization procedures, which do not rely on this assumption. In addition, without a measure of perceived rotation, we cannot rule out the possibility that some sort of weighted averaging of the parallax and disparity signals also takes place, as implied by Expression (3).

20 71 6 ROGERS AND COLLETT The major implication of the results reported here is that motion parallax should not be thought of as a cue that only signals the amount of depth present. Parallax transformations provide information about both the structure and movement of 3-D objects and surfaces. The way in which a given motion parallax transformation is interpreted will depend, in many cases, on the presence or absence of other cues, such as disparity, perspective, and texture gradients. The apparent rotation of an inverted hollow face or a projected random dot stereogram with movement of the observer both testify to this fact (Lindsay & Norman, 1972; Rock, 1983). Given apparently contradictory disparity and parallax information about the structure of 3-D objects, it would appear that the visual system attempts to find a solution that is maximally consistent with all the available information. REFERENCES Adelson E. H. (1985). Rigid objects that appear highly non-rigid. Znvesiigative Ophthalmology and Visual Science, 26, Braunstein, M. (1966). Sensitivity of the observer to transformations of the visual field. Journal of Experimental Psychology, 72, Braunstein, M. (1968). Motion and texture as sources of slant information. Journal of Experimental Psychology, 78, Bruce, V. & Green, P. (1985). Visual perception. London: Lawrence Erlbaum Associates Ltd. Buckley, D., Frisby, J., & Mayhew, J. (1988). Interaction of texture and stereo cues in the perception of surface slant. Perception, I7 (Abstract). Graham, M. E. & Rogers, 3. J. (1982). Simultaneous and successive contrast effects in the perception of depth from motion and stereoscopic information. Perception, 11, Graham, N. & Nachmias J. (1971). Detection of grating patterns containing two spatial frequencies. Vision Research, 11, Harris, J. (1980). How does adaptation to disparity affect the perception of reversible figures? American Journal of Psychology, 93, Helmholtz, H. von (1924). Physiological optics (edited by J.P.C. Southall). New York: Optical Society of New York. Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception and Psychophysics, 14, I I. Julesz, B. (1971). Foundations of cyclopean perception. Chicago, IL: University of Chicago Press. Koenderink, J. J. (1985). Space, form and optical deformations. In D. J. Ingle, M. Jeannerod, & D. N. Lee, Brain mechanisms and spatial vision. Dordrecht: Martinus Nijhoff. Koenderink, J. J. (1986). Optic flow. Vision Research, 26, Lindsay, P. & Norman, D. (1972). Human in~ornialion~ruce~si?zg. New York: Academic Press. Marr, D. (1982). Vision. San Francisco, CA: Freeman. Richards, W. (1985). Structure from stereo and motion. Journal qf [he Optical Society of America, 2, Rock, I. (1983). The logic of percepfion. Cambridge, MA: M.I.T. Press. Rogers, B. J. & Collett, T. S. (1985). Rigidity and perceived depth in surfaces specified by disparity and parallax cues. Investigative ophthalmology and visual science, 26, 242 (Supplement). Rogers, B. J. & Graham, M. E. (1979). Motion parallax as an independent cue for depth perception. Perception, 8,

21 MOTION PARALLAX AND BINOCULAR DISPARITY 71 7 Rogers, B. J. & Graham, M. E. (1982). Similarities between motion parallax and stereopsis in human depth perception. Vision Research, 22, Rogers, B. J. & Graham, M. E. (1983). Anisotropies in the perception of three-dimensional surfaces. Science, 22I, Rogers, B. J. & Graham, M. E. (1985). Motion parallax and the perception of three-dimensional surfaces. In D. J. Ingle, M. Jeannerod, & D. N. Lee (Eds.), Brain mechanisms and spatial vision. Dordrecht: Martinus Nijhoff. Rogers, B. J., Ono, H., & Rogers, S. (1988). The role of visual and non-visual information in disambiguating motion parallax transformations. Investigative ophthalmology and visual science, 29, 265 (Supplement). Rogers, B. J. & Koenderink, J. J. (1986). Monocular aniseikonia: A motion parallax analogue of the disparity induced effect. Nature, 332, Sekuler, R. & Blake, R. (1985). Perception. New York: Knopf. Ullman, S. (1979). The interpretation of visual motion. Cambridge, MA: M.I.T. Press. Wallach, H. & OConnell, D. (1953). The kinetic depth effect. Journal of Experimental Psychology, 45, Wheatstone, C. (1838). On binocular vision; and on the stereoscope, an instrument for illustrating its phenomena. Report of the British Association, Transactions of the Sections, 16-17, Revised manuscript received 6 June I989