An integrated system for virtual scene rendering, stereo reconstruction, and accuracy estimation.

Transcription

1 An integrated system for virtual scene rendering, stereo reconstruction, and accuracy estimation. Marichal-Hernández J.G., Pérez Nava F*., osa F., estreo., odríguez-amos J.M. Universidad de La Laguna, Facultades de Física y Matemáticas. {jmariher@ull.es, fdoerez@ull.es, frosa@ull.es, rrestre@iac.es, jmramos@ull.es} Abstract his article deicts an integrated system caable of generating virtual stereo scenes, reconstructing a threedimensional ma of distances, and calculating the accuracy of the results obtained. his tool is articularly useful for verifying the validity of distance reconstruction techniques, and is robust enough to use in a wide variety of scenarios since it avoids the need for an imagecature system in laboratory simulations. he imortance of the distance reconstruction roblem lies in the range of its alications: industrial insection and quality control, surveillance and security, autonomous vehicles, robotic systems, medical image analysis, image databases, virtual reality, teleresence, and telerobotics. he system is based on the VML standard for three-dimensional geometric scene reresentation, on OenGL for rendering virtual scenes, and on the W algorithm for the roblem of three-dimensional reconstruction. Keywords- 3D modeling, rendering, stereo reconstruction, stereo accuracy and validation. 1. Introduction he goal of 3D scene reconstruction is to take a set of images and estimate the ositions and orientations of the cameras that roduced the images, as well as a reresentation of the scene that was imaged. his is an examle of an inverse roblem named the stereo reconstruction roblem. he forward (or direct) roblem is: given a scene and the osition and orientation of a set of cameras, what is the exected image? his is the area of comuter grahics known as rendering [1]. While both roblems have their own difficulties, it is widely believed that the inverse roblem is considerably harder than the direct roblem. ecent advances in the develoment of efficient otimization algorithms [2] have roduced a dramatic imact on the stereo reconstruction roblem, according to several benchmarks that evaluate stereo algorithms using real scenes with known reconstructions [3]. Almost all to-erforming methods aroach the 3D reconstruction roblem from the oint of view of energy function otimization. In our system we rovide a solution for both the direct (rendering) and inverse (stereo reconstruction) roblem. his allows us to estimate the accuracy and robustness of the inverse solution. he need for a tested solution to the stereo roblem is of great imortance for several alications in diverse fields. One of these alications is the Virtual Acoustic Sace (VAS). he VAS is a multidiscilinary roblem that aims to model a three-dimensional sace by just using sounds. he roject s goal is to cature a 3D scene via cameras and to obtain a set of elements, the acoustic rojection of which will give the illusion of having the objects covered by a thin, sound-roducing layer. A world of virtual sounds could thus be generated which directly corresonds to the real world. Achieving the objective of the VAS requires techniques in artificial vision, 3D reconstruction, and sound satialization. he 3D reconstruction of a scene yields elements such as distance mas, in which the sound sources are laced at given distances by using sound satialization. he technique of satializing sounds by way of a transfer function related to the subject who receives the sounds, called a Head-elated ransfer Function (HF), is sufficiently advanced to assure roer satial ercetion. Closely related with the VAS is the VSIDS (Virtual Scene Immersive Distance Simulator) module, which solves the direct roblem thus allowing for the substitution of the image cature system by a simulation system which ermits for easier testing of different scenes and for imroved 3D reconstruction, sound generation, and satialization. he simulator can be used to erform exeriments that clarify both the audio channel s caacity to transmit the information of the scene s three-dimensional structure, as well as the ability of the emloyed stereo algorithm to resolve the inverse roblem by measuring the degree of recision in retrieving the 3D distances. In this aer we resent the results of such exeriments, making use of stereo reconstruction techniques to calculate the distance ma. Figure 1 shows a comlete diagram of the modules already imlemented. he W [4] algorithm, currently one of the most advanced, is used for the stereo roblem. *Corresonding author

2 his algorithm can also be generalized to multiview stereo. Its arallel nature lets it erform real-time image samling as long as the hardware chosen is adequate, for examle a Grahics Processing Unit (GPU). 2. he VSIDS Module he VSIDS simulator as a whole consists of a system for acquiring the osition and orientation of the head of the user to which it is connected. his information is then used to osition a virtual camera in a virtual scene, as required by the exeriment being erformed. he simulator allows for a measurement of distance, color, and outline arameters for objects located within the field of view of the camera. When the virtual scene is made to match the reality that surrounds the user, while at the same time synchronizing his movements, the result is an immersive virtual reality system. he information outut by the simulator is then used to generate the satiated sounds that the user hears. he goal of this feedback is to achieve acoustic immersion without transmitting to the user any visual information. he elements handled by the simulator are a virtual world on one hand, and the real world osition and orientation of the user s head on the other. his module merges both elements, rendering with a virtual camera located within a virtual scene that which a real camera in the same osition would see. he rendering does not yield what might be called a real-world image. Additionally, the renderer must suly distance information, distances measured from the camera to the objects, as well as a geometric outline of the edges in the world. Since the communications channel with the user must be necessarily reduced, a strategy must be secified for selecting the most relevant information before it is sent to the generator. his selection takes lace when deciding on a stereo-ixel or sound-voxel strategy. he next module, the sound generator, is caable of roducing in the user the sensation that a sound roceeds from a secific oint in sace. If a ixel is an image element in the two-dimensional sace reresented on a screen, then a stereo-ixel is a sound element in the three-dimensional sace of the user s acoustic ercetion. he stereo-ixel selection strategy reconstructs, calculates, and selects the attribute information generated by the simulator module which is to be sent to the next block. 2.1 Scene Loader (SL). Of fundamental imortance to the simulator is the otential and versatility it is given for reresenting the three-dimensional geometry of the worlds it simulates. hese requirements hinge on two oints: the reresentation of the world in the simulator s memory, and its file reresentation while keeing to a geometric descrition standard. he choice of a standard must conform to the needs of both the simulator and the creator of the virtual worlds, a 3D designer used to working with the highly secialized software roducts develoed for this task: 3DStudio, LightWave, AutoCad, or Blender, among others. o this end we have imlemented, after considering different otions, a VML world loader, subject to the VML 97 norm (he Virtual eality Modeling Language International Standard ISO/IEC :1997). Said module will let us maniulate the geometry exorted by the different 3D design latforms. Among the characteristics of the module imlemented stand out the ability to organize and subdivide files (Inline nodes), geometric object reutilization (DEF-USE nodes), and triangular and square color geometry suort with surface or node normals (ransform-shae-geometry Indexed Face Set nodes). his module, combined with the one for hardware accelerated three-dimensional reresentation described later, achieves high-resolution interactive frame rates in excess of 18 frames er second in scenes with one million vertices Virtual world rendering (VW) Once a geometric reresentation of the virtual world is available in memory, the stes needed to immerse the user in the simulated environment require registering his osition in the virtual world, adjusting this measurement to the scale of the virtual world, and lacing a camera in the virtual world. Put another way, the role of the user in the virtual world is layed by the camera in front of which the world is rojected. his camera follows in the virtual scene the user s movements in the real world. For rendering the virtual world, we have chosen the OenGL standard for three-dimensional hardware acceleration. o imrove erformance we have used the advanced techniques contained in secification 1.3 of the standard (OenGL 1.3 Secification), secifically the DrawElements extension along with generation lists, which allows for interactive maniulations in frames containing u to one million vertices. Adequately formatted information from the world, as well as the camera s osition, must be constantly relayed to the API OenGL of the grahics card. he hardware-software latform consists of a ViewPoint nvidia Geforce 7800 SE grahics card and an AMD with 1 GB of memory, running the GNU/Linux Debian 3.0 OS. he world reresented is generated by the LightScae utility in test mode. Even if the worlds needed for exeriments are not necessarily as sohisticated and comlex as the ones shown, the utility is ready to suort highly elaborate models Distance extractor (DE). What distinguishes the simulator we imlemented from other virtual world viewers is the emhasis it laces on acquiring the distances from the objects to the oint of

3 observation. o obtain these measurements we have resorted to state of the art 3D grahics acceleration hardware, which yielded two methods for maniulating distances. he first method, based on accessing the internal Z-buffer of the OenGL machine, is slower but comatible with the majority of grahics cards on the market. he second method is based on the rogramming of the more advanced GPUs (Grahics Processing Units), only available on the most recent cards, such as the GeForce4 i. With the first method, interactive distance extraction is only ossible with the simlest scenes. For more comlex worlds, this method of distance extraction is only viable u to 10 frames er second. Using higher caacity hardware, however, allows for higher interactivity rates within comlex sceneries. SL (Scene Loader) VSIDS VW (ender) SP (Stereo Pair) SS Comuted Distances) Figure 1: General Scheme. 3. Stereo Scene econstruction: he SS module. he distance mas generated by VSIDS are very useful for testing the viability of the user s acoustic immersion in the 3D scene under ideal circumstances. However, the distance ma reconstruction in VSIDS is obtained by resolving the direct roblem when in fact, for the acoustic immersion to be of ractical use, the distance ma must be obtained by resolving the inverse roblem (obtaining the geometric model based on the images of the scene). his inverse roblem is the stereo roblem, of great imortance in comuter imaging. o solve this roblem we could use several ublicly available stereo algorithms. However, in our system we have imlemented the ree eweighed Message Passing (W) [4] algorithm. here are two main reasons for this selection: first, it outerforms other well known stereo algorithms based on Looy Belief Proagation or Iterated Conditional models [2]; and second, it can be aralellizable if real time erformance is needed. 3.1 Stereo reconstruction with the W algorithm. DE (rue Distances) SACC (Accuracy) In this section we will resent an overview of the W algorithm. Given a stereo air comosed of two images I and I', the stereo roblem can be solved by assigning to every ixel on image I of the stereo air a disarity value (roortional to the inverse of the 3D distance), which we write as d. his disarity is equal to the difference in image coordinates between ixel in I and its corresonding ixel in the other image I' of the stereo air. herefore, to obtain the disarity it is necessary to obtain for each ixel in I its corresonding ixel in the other image I'. his corresondence roblem can be solved by a variety of aroaches. State of the art methods solve this corresondence roblem by obtaining the minimum of an energy function E. his energy function E, which can also be viewed as the log likelihood of the osterior distribution of a Markov andom Field (MF) [5], is comosed of a data energy Ed and smoothness energy Es, E = Ed+λEs, where the arameter λ measures the relative imortance of each term. he data energy is simly the sum of a set of er-ixel data costs c(d), Ed = Σ c(d). In the MF framework, the data energy comes from the (negative) log likelihood of the measurement noise. We assume that ixels in the image form a 2D grid, so that each can also be written in terms of its coordinates = (i,. If we use the standard 4-connected neighborhood system, then the smoothness energy is the sum of satially varying horizontal and vertical nearest-neighbor smoothness costs, Vq(d, dq), where if = (i, and q = (s, t) then i - s + j - t = 1. If we let N denote the set of all such neighboring ixel airs, the smoothness energy is: E s = V q {, q} N In this aer we will use: ( d, d ) (1) q c(d)= I()-I'(+d) 2, where I() and I'(+d ) denote image intensities in ixel on image I and ixel +d on image I' (as vectors due to the use of color images). Vq(d, dq)=0 if d dq Vq(d, dq)=1 if d=dq. ree-reweighed message assing (W) [4] is a message-assing algorithm that can be used to obtain an aroximate minimum of the energy function E. t Let M q be the message that ixel sends to its neighbor q at iteration t; this is a vector of size m (the number of ossible disarities). he message udate rule is: M t q µ q( c ( d ) + M ( d ) = min s N ( ) q d q t 1 M q ( d q) + V q( d t 1 s, d ( d q ) q ) (2) he coefficients µq are determined as follows: first, a set of trees from the neighborhood grah (a 2D grid in our case) is chosen so that each edge is in at least one tree; then a robability distribution ρ over the set of

4 trees is chosen; finally, µq is set to ρq/ρ, i.e. the robability that a tree chosen randomly under ρ contains edge (, q) given that it contains. An interesting feature of the W algorithm is that for any messages it is ossible to comute a lower bound in the energy E. he original W algorithm used in this aer does not necessarily converge, and does not, in fact, guarantee that the lower bound always increases with time. hese roblems can be solved using daming in the W udates or by the use of a sequential version of the W. [6] 4 Stereo Accuracy Estimation: he SACC module In this section we describe the quality metrics we use for evaluating the erformance of the stereo corresondence algorithm in the simulated scenes. wo general aroaches for validating stereo algorithms are to comute error statistics with resect to some ground truth data, or to evaluate the synthetic images obtained by waring the reference or unseen images by the comuted disarity ma. Since the VSIDS module rovides us ground truth data of the scene, we have chosen the first aroach. Following [3], we comute two quality measures based on known ground truth data: 1. elative MS (root-mean-squared) error (measured in distance units) between the comuted distance ma zc(i, and the ground truth ma z (i,, i.e., 1 N = ( i, ( z ( i, z ( i, ) C zmax 2 1/ 2 (3) where N is the total number of ixels and zmax is the maximum true distance in the scene. by the VSIDS module is shown in Figures 2 and 4 resectively. Several difficulties arise when any stereo algorithm tries to find the correct reconstruction in both cases: there are many textureless regions (walls, floors) where it is very difficult to locally solve the corresondence roblem, so the smoothness constraint has to be enforced. By contrast, there are abrut discontinuities in the distance ma (chairs, armchairs, table) where the smoothness constraint fails. Note also that these abrut changes usually coincide with occluded oints. Both image airs were generated as GB 256x256 images. No color re-rocessing was done and the stereo algorithm used the three comonents of the GB image. 4.2 Accuracy esults Once the image air was generated we reconstructed the distance ma with the W algorithm. In the "armchair" scene the disarities were in the range [-34, -6]. For the "table" scene the disarities were in a wider range: [-57, -8]. As distances are inversely related to disarities, an effect of the discrete nature of the disarities is that the distance ma is more finely samled at near distances and more coarsely samled at far distances. his quantization effect is clearly shown in the resented inverse distance mas. he true and comuted distance mas for the armchair and table scenes are shown in Figures 3 and 5. Numerical results for the quality measures for otimal values of λ (see Section 4.3) are given in able 1. he distribution of errors is shown in Figures 6 and Percentage of bad matching ixels, ( z ( i, z ( i, zmax ) 100 B = C > δ N (4) ( i, where δ is a relative error tolerance. For the exeriments in this aer we use δ =. Figure 2: he armchair stereo air 4. Exerimental esults. In this section, we describe the exeriments used to evaluate the individual modules of our system. Our main interest is to test the coherence between the direct and inverse estimations of the distance ma. 4.1 est data o evaluate the system in comlex environments we have selected two realistic indoor scenes. he first scene will be named the armchair scene and the second the table scene. he stereo air rendered for both scenes Figure 3: Distances recovered for the armchair stereo air by the direct (rendering) and inverse (stereo reconstruction) methods.

5 6 4 Figure 4: he table stereo air Figure 7: Error (direct estimation minus inverse estimation) histogram for the table stereo air Figure 5: Distances recovered for the table stereo air by the direct (rendering) and inverse (stereo reconstruction) methods. esults show that a reliable 3D reconstruction is obtained with the W method. elative MS error for the table scene is very small and slightly larger for the armchair scene. he ercentage of misclassified ixels is again very small for the table scene and larger for the armchair. However, reconstruction errors in the armchair scene are biased by distance errors in the front wall, while distances near the user are well reconstructed. 4.3 Otimal values for the arameters Scene λ B "Armchair" % "able" % able 1: Quality measures for the stereo reconstruction Another tye of exeriment tried to determine the otimum value for λ in the energy function E = Ed+λEs. o obtain this otimum we comuted the quality measure for several values of λ. In Figures 8 and 9 we can see a lot of the measure versus λ. Otimum values were λ =6.0 and λ=5.1 for the "armchair" and "table" scenes resectively. "Armchair" scene: elative error versus relative frequency 3 "Armchair" scene: lambda versus 1 elative frequency elative error Figure 6: Error (direct estimation minus inverse estimation) for the armchair stereo air lambda Figure 8: Determination of the otimum value for l in the armchair stereo air. Plot of l versus the quality measure.

6 "able" scene: lambda versus "able" scene: Number of iterations versus lambda Number of iterations Figure 9: Determination of the otimum value for l in the table stereo air. Plot of l versus the quality measure. As we can in Figures 8 and 9 there is a wide range of λ values that roduce aroximately the same values for the quality measure in both scenes. his is highly desirable since the comutation of the otimal λ value has a high comutational cost. In fact, Figures 8 and 9 were obtained running the W algorithm for several artitions of the ossible values of λ in a cluster of 64 Intel Itanium 2 rocessors at the SAII Service of the University of La Laguna (ULL). he last tye of exeriment tried to determine the otimum value for the number of iterations in the W algorithm. o obtain this otimum we comuted the quality measure at each iteration of the algorithm for the otimum value of λ. In Figures 10 and 11 we can see a lot of the measure against the number of iterations. he otimum values for the number of iterations nit were nit = 13 and nit =16 for the "armchair" and "table" scenes resectively. Figure 11: Plot of the number of iterations versus the quality measure in the table stereo air. 5 Conclusions and future extensions We have imlemented a robust tool for testing different stereo algorithms and scenarios running on owerful electronic hardware, namely, a GPU where real time develoments and high samle rates can be successfully rocessed. he good results shown here can be imroved by generalizing the stereo algorithm for multi-view stereo vision. Even more realistic scenes and added textures can be easy simulated by using GPUs. Acknowledgements his work has been artially funded by Programa Nacional I+D+i (Projects DPI and IC ) of the Ministerio de Ciencia y ecnología, by the Euroean egional Develoment Fund" (EDF), and by Becas Convenio Cajacanarias- ULL (2005). We thank the SAII Service of the ULL for the use of its comuting facilities and to Sergio Ortiz Eirín for a critical reading of this aer. eferences 3 1 "Armchair" scene: Number of iterations versus Number of iterations Figure 10: Plot of the number of iterations versus the quality measure in the armchair stereo air for the otimal l value. [1] J. Foley, A. van Dam, S. Finer, and J. Hughes. Comuter Grahics, rinciles and ractice. Addison-Wesley, 2nd ed. edition, [2]. Szeliski,. Zabih, D. Scharstein, O. Veksler, V. Kolmogorov, A. Agarwala, M. aen, C. other. "A Comarative Study of Enegy Minimization Methods for Markov andom Fields'', unublished [3] D. Scharstein and. Szeliski. A axonomy and Evaluation of Dense wo-frame Stereo Corresondence Algorithms. IJCV 47(1/2/3):7-42, Aril-June 2002 [4] M. Wainwright,. Jaakkola, A. Willsky, W belief roagation and aroximate ML estimation by seudomoment matching. In: AISAS. (2003) [5] S. Geman, D. Geman: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE rans Pattern Anal Mach Intell 6 (1984) [6] V. Kolmogorov and M. Wainwright. On otimality of tree-reweighted max-roduct message-assing. Uncertainty in Artificial Intelligence, July 2005, Edinburgh, Scotland.