3D Reconstruction for Rendering

Transcription

1 3D Reconstruction for Rendering Topic in Iage-Based Modeling and Rendering CSE9 J00 Lecture 6 This lecture [DTM96] Paul E. Debevec, Caillo J. Talor, Jitendra Malik, Modeling and Rendering Architecture fro Photographs: A hbrid geoetr and iage-based approach. Technical report UCB/CSD , Universit of California at Berkele, 996. [TK94] Caillo J. Talor and David J. Kriegan Structure and Motion fro Line Segents in Multiple Iages. IEEE Trans. Pattern Anal. Machine Intell. 7() Noveber 995.

2 Introduction Presents an approach for odeling and rendering eisting architectural scenes fro sparse sets of still photographs. Geoetr-based ethods: odeling progra is used to constructing odel. Drawbacks: labor-intensive, difficult to verif and ost of all: unrealistic. Iage-based: creating odel directl fro photographs. Relies on stereo algoriths, has a lot constrains on the input. Hbrid approach Cobine the strengths of both geoetrbased and iage-based ethods. Src [DTM96]

3 Hbrid approach Photograetric odeling: geoetric odel of the architecture is recovered interactivel. View-dependent teture apping: create novel view. Model-based stereo: additional geoetric detail can be recovered. Eaple Src [DTM96]

4 Photograetric odeling Scene represented as a constrained hierarchical odel of paraetric priitives (block). Relationships between blocks are represented b a rotation atri and a translation vector. Source [DTM96] Representation of block paraeters as sbol references. A single variable can be referenced b the odel in ultiple places, allowing constraints of setr to be ebedded in the odel. Advantages. Advantages of odeling the scene with blocks: Most architecture can be readil decoposed into a set of blocks. Blocks iplicitl odel coon architectural constraints. Convenient to anipulate block priitives. Surface of the scene is readil obtained fro the blocks. Reduce the nuber of paraeters need to recover.

5 Reconstruction Based on the reconstruction of infinite straight lines. [TK94] Model paraeters and caera positions are coputed b iniizing objective function Ο. n j= i= ( F p q u ) O = Error ), ( i, j ij Feature i in iage j. Disparit between the actual and epected iage easureents Caera position of iage i. Structure of feature j. Geoetr of Straight Lines A straight line is represented b a tuple <v,d>. The line and the origin define a plane whose noral is vector The edge in the iage will be: + Y + z = 0

6 Projection Function F d v t R F ),,, ( The Error Function ( ) ( ) z z h h l = = + = Error function represents disparit between the actual and epected iage easureents. l h h s h s h ) ( + =

7 The Error Function <cont.> The total error between the observed edge segent and the predicted edge as: l Error = h ( s) ds = ( h + hh 0 3 T T = ( A BA) + h ) A = B = 3 l ( + ) This is a projection function of <R j,t j,v i, d i > Recover Algorith - Overview Miniizing object function O with respect to <R j,t j,v i, d i > Estiates for caera rotations R j. Estiates for caera translations t j and the paraeters of the odel <v i, d i >. Non-linear iniization over the entire paraeter space.

8 Estiate R j Initial estiates for R j can be entered anuall. Can be re-estiated after v i are estiated. Estiate v i Based on constraints c T c w ( R v) = 0 w Use easured noral to the plane passing through caera center and the observed edge c ' =

9 Estiate v i <cont.> n n c T c w T ( w R v) = 0 C = ( ' ij R jvi ) = i= j= i= C Ai v i is deterined b iniizing each C Ai with respect to v i. n degrees of freedo. Iprove the estiates of R j, v i b iniizing C with respect to (R j, v i ). n+3(-) degrees of freedo. Estiates d i and t j Again, fro constraint: c T c w w w ( w R ( d t c ) = 0 d i and t j are estiated to iniize the objective function: C n T ( R j ( di t j ) ij = ' j= i=

10 Estiates d i and t j Note that v i is orthogonal to d i. v v = 0 i i v v = 0 i i v v i i = 0 d i = α v + βv i i C n n T T ( ' CR j= ( α ( v + βvˆ ' Rt j ( d i t j ) ) ij i i ij ˆ t ˆ tzz = ˆ j= i= j= i= Then, closed for linear least squares can be applied to obtain estiates for these paraeters. Miniizing the Objective Function. n j= i= ( F p q u ) O = Error ), ( i, Using variant of the Newton-Raphson ethod. Optiization requires fewer than ten iterations. Edge of the recovered odels tpicall confor to the original photographs to within a piel. j ij

11 Si iages taken fro a sequence of 4 taken fro a section of Yale Center for Sstes Science office cople Source [TK96] A set of views of the reconstruction of the scene. The sall coordinate aes in figure d represent the reconstructed caera positions. Above: Three of twelve photographs used to reconstruct the entire eterior of Universit High School in Urbana, Illinois. The superiposed lines indicate the edges the user has arked. Left: The high school odel, reconstructed fro twelve photographs. Aerial view showing the recovered caera positions.

12 The edges of the reconstructed odel, projected through the recovered caera positions and overlaid on the corresponding iages. The recovered odel confors to the photographs to within a piel in all twelve iages, indicating that the building has been accuratel reconstructed. View-Dependent Teture- Mapping Projecting the original photographs onto the odel based on viewing frustu. Single-iage projection: iage-space shadow ap.

13 Coposition of ultiple iages Need ultiple iages in order to render the entire odel. Copositing ultiple iages. Use the one with viewing angle closest to that of rendering view. Projected iage weights are coputed at ever piel of ever projected rendering. In fact, a single weight is used across a flat surface. Src [DTM96]

14 Blending iages For piel B, both views are equall good. One solution would be to use piel values fro view to the left of B and piel values fro view to the right of B. A better solution is to blend piels fro view and view in the area between A and C according to their relative fitness values. Reoval of obstructions. Unwanted objects in the original photograph a be projected onto the surface of the odel. Mask out these objects with a reserved color. A photograph in which the user has asked out two interposed buildings that obscure the architecture of interest. These asked regions will consequentl not be used in the rendering process.

15 Filling a holes. Unfilled piels on the peripher of the hole are successivel filled with the average values of the neighboring piels. After each iteration, the region of unfilled piels is eroded b a width of one piel. Universit High School flaround, with trees. Universit High School flaround, without trees. For this sequence, the trees were asked out of the twelve original photographs of the building. The asked sections were then filled in autoaticall b the iage coposition algorith.

16 Model-Based Stereo Soe geoetric detail is not captured in the odel. Measures how the actual scene deviates fro the approiate odel. The odel serves to place the iages into a coon frae of reference. Traditional stereo Given iages (ke and offset) odelbased stereo coputes the associated depth ap for the ke iage b deterining corresponding points in the ke and offset iages. When the ke and offset iages are taken fro relativel far apart, corresponding piel neighborhood can be foreshortened ver differentl.

17 Warp offset Project the offset iage onto the odel and view it fro the position of the ke iage. (warped offset). Piel neighborhoods are copared between the ke and warp offset iages. Src [DTM96] Advantage of warp offset. Reduction of differences in foreshortening. An point in the scene which lies on the approiate odel will have zero disparit between the ke iage and the warped offset iage. Eas to convert to a depth ap for the ke iage. Less sensitive to noise in iage easureents.

18 Epipolar Geoetr Src [DTM96]

19 Result of odel-based stereo. Novel views of the scene generated fro four original photographs. The depth is coputed fro odel-based stereo and the fraes are ade b copositing iage-based renderings with view-dependent teture-apping. Result The Capanile The Capanile, a short fil shown at the SIGGRAPH'97 Electronic Theatre, used Façade to construct a photorealistic odel of Berkele's clock tower and the surrounding capus

20 Result San Francisco Museu of Modern Art b Yizhou Yu Iproveent of Façade. 3-D Model Reconstruction of thetaj Mahal using Façade. George Borshukov

21 Coercial use. Seven photographs of a street scene in downtown Palo Alto, northern California. Modeling tie 3 hours. Source: Past, present and future Coercial products: Canoa, REALVIZ IageModeler, PhotoModeler Pro 4.5 Mane entertainent: The Matri. Using architectural conventions and practical considerations to enhance the reconstruction odel. Reconstruct fro single, uncalibrated iage. And ore