Motivation. TensorTextures: Multilinear Image-Based Rendering. Image-Based Rendering. Our Contribution. BTF Texture Mapping [Dana et al.

Transcription

1 Motvaton ensoretures: Multlnear Image-Based Renderng Computer Graphcs Goal: Generaton of photorealstc vrtual envronments Classcal Computer Graphcs: Model based Renderng From obect models to mages Model specfes geometry of a scene and surface propertes are generated by proectng model onto an mage plane and computng surface shadng Photorealsm requres comple models ffcult me consumng Image-Based Renderng [Gortler et al. 996, Levoy Hanrahan 996, ebevec, aylor Malk 996, ] World s modeled by a collecton of mages (and possbly some coarse geometry) hese mages are used to synthesze novel mages representng the scene from arbtrary vewponts and llumnatons Advantages: Renderng s decoupled from the scene complety Photorealsm s mproved Our Contrbuton We ntroduce a tensor framework for mage-based renderng (IBR) Specfcally, renderng of tetured surfaces Surface appearance s determned by the comple nteracton of multple factors: Scene geometry Illumnaton Imagng Bdrectonal eture Functon BF eture Mappng [ana et al. 999] BF: Captures the appearance of etended tetured surfaces wth Spatally varyng reflectance Surface mesostructure ( teture) Subsurface scatterng Etc. Generalzaton of BRF, whch accounts only for surface mcrostructure at a pont Standard eture Mappng BF eture Mappng Concrete Pebbles Plaster

2 BF Reflectance as a functon of poston on surface, vew drecton, and llumnaton drecton f BF ( v v, y, θ, φ, θ, φ ) poston on surface vew drecton llumnaton drecton photometrc angles he BF captures shadng and mesostructural self-shadowng, self-occluson, nterreflecton ensoreture Mappng Geometry + eture Standard eture Mappng ensoreture Mappng ensoretures: Learns BFs from ensembles of sample mages onlnear generatve BF model Background BF ntroduced by ana et al. [999] BF acquston devces [ebevec et al. 000] [ana 00] [Furukawa et al. 00] [Han Perln 00] (BF Kaledoscope) BF based renderng methods Polynomal teture maps [Malzbender et al. 00] Synthess of BFs for curved surfaces [Lu et al. 00] [ong et al. 00] ensoretures Overvew. Mathematcal foundatons: Egentetures Lnear Analyss / Prncpal Components Analyss fed vewpont, changng llumnaton changng vewpont and llumnaton. ensoretures onlnear (multlnear) Analyss / ensor decomposton. Eperments and results HE FOLLOWIG PREVIEW HAS BEE APPROVE FOR ALL AUIECES BY HE MOIO PICURE ISASSOCIAIO OF AMERICA

3 Smple ata Acquston: Fed Vewpont, Varyng Illumnaton Egentetures PCA (Matr Algebra) Sample mages are ponts n pel space IRM Pels teel M IRM he -Mode Case (fed vewpont, varyng llumnaton) teel teel Egentetures captures varaton across llumnatons Prncpal Components Analyss (PCA) Egentetures teel M teel M Image Representaton usng PCA teel d = Uc d c c teel teel teel c0 Egentetures captures varaton across llumnatons + c An arbtrary mage + c Its PCA Representaton + ck ote: hs s a lnear representaton

4 Samplng Multple Vewponts and Image Rectfcaton Rectfy hs poses a -mode BF estmaton problem Vewpont, llumnaton, and pel modes Rectfyng Homography Applyng PCA Image unwarpng p ' = Hp H can be computed gven at least 4 fducals p p Rectfy 4 4 Pels Egentetures varaton across vews and llumnatons Rectfy PCA Reconstructon orgnal bass vectors bass vectors ensoretures (ensor Algebra)

5 System agram ensoretures: -Mode ata ensor Image Acquston, Pre-processng Organzaton hs leads to a multlnear BF learnng method System agram ensoreture: -Mode ata ensor Image Acquston, Pre-processng Organzaton ensor ecomposton mensonalty Reducton Background on ensor ecomposton Matr ecomposton - SV Factor Analyss: Psychometrcs, Econometrcs, Chemometrcs, SV: [Beltran, 87] (Gornalle d Matematche ) Sulle funzon blnear Uvews Ullums [Eckart and Young, 96] (Psychometrka) he appromaton of one matr by another of lower rank -Way Factor Analyss: [ucker,966] (Psychometrka) Some mathematcal notes on three mode factor analyss [Kroonenberg and e Leeuw, 980] -mode ALS -Way Factor Analyss: [Kapteyn, eudecker, and Wansbeek, 986] -way ALS factor analyss [Franc, 99] tensor algebra [ens horne, 989] [de Lathauwer, 997] A matr IR II has a column and row space SV orthogonalzes these spaces and decomposes = U SU ( U contans the egentetures ) Rewrte n terms of mode-n products = S U U 5

6 ensor ecomposton s a n-dmensonal matr, comprsng -spaces -mode SV s the natural generalzaton of SV = U SU = S U U ensoreture ecomposton -mode SV orthogonalzes these spaces decomposes as the mode-n product of -orthogonal spaces = Ζ... U U U U 4 Z core tensor; governs nteracton between mode matrces U n mode-n matr, s the column space of (n) = Z U U U teels llums vews ensor ecomposton -Mode SV Algorthm U U Z U. For n=,,, compute matr U n by computng the SV of the flattened matr (n) and settng to be the left matr of the SV. U n vec =Z R = r = U U U teels llums. vews R r = R r = σ r r r u teels, r ou llums., r ( ) = ( U U U ) vec( Z ) vews llums teels ou vews, r. Solve for the core tensor as follows Z = U U L U Computng U vews (vews) -flatten along the vew pont dmenson U vews orthogonalze the column space of (vews) (vews) (llums) -flatten along the llumnaton dmenson U llums orthogonalzes the column space of (llums) (llums) Computng U llums Illums

7 Computng Uteels -Mode SV Algorthm (teels). For n=,,, compute matr Un by computng the SV of the flattened matr (n) and settng Un to be the left matr of the SV.. Solve for the core tensor as follows Z = U U (teels) - flatten along the pel dmenson L U Uteels orthogonal column space of (teels) egenmages Mode- Product Mode- Product Mode-n product s a generalzaton of the product of two matrces It s the product of a tensor wth a matr Mode-n product of B IR I...I n. J n I n+..i A IR I...I n...i and M = IR J B M I = I J A I M ℜ J n In J I J I mode-n product: JI I (A nm )... A ℜ I I L I -th order tensor matr (-nd order tensor) JI I I I n JI B = A nm I n = a... m nn n n n +... n n n +... n ensoretures: = Z Upels B = A n M where B( n) = M A (n ) ensoretures: = Z Upels ensoretures: eplctly represent covarance across factors Varaton n Varaton n Vewng recton V Va r ew a ng to n n re ct on Varaton n ensoretures: eplctly represent covarance across factors 7

8 ensoretures vs. PCA Strategc mensonalty Reducton Multlnear Analyss / ensoretures: = Z U U U teels llums. vews Lnear Analyss : ( ) = Z(teels) U vews Ullums. (teels) 44 U teels data matr bass matr coeffcent matr ensoretures subsumes PCA / Egentetures Strategc mensonalty Reducton #bas s PCA - Egentetures ensoretures 7 System agram Computng v new : Homogeneous Barycentrc Blend V Image Acquston, Pre-processng Organzaton? t t k V new t V ensor ecomposton mensonalty Reducton Geometry Vewpont Illumnaton U vews U llums? Renderng Algorthm V new t + V t + = V V t t + + k t = k t k V k t = ( V Vnew ) ( Vk Vnew )) ( V V ) ( V V )) k

9 Synthess Algorthm / eture Representaton d = Rendered eture for a Planar Surface lnew vnew llumnaton vewpont Rendered etures for Cylnder Renderng on Arbtrary Geometry ensoretures renderngs v ew Bonn natural BF datasets = v v teels d llum. l l Vdeo Flntstones Brd Scarecrows Quarterly reasure Chest 9