Goal. Rendering Complex Scenes on Mobile Terminals or on the web. Rendering on Mobile Terminals. Rendering on Mobile Terminals. Walking through images

Transcription

1 Goal Walking though s Kadi Bouatouch IRISA Univesité de Rennes I, Fance Rendeing Comple Scenes on Mobile Teminals o on the web Rendeing on Mobile Teminals Rendeing on Mobile Teminals IBR methods: endeing of comple scenes on a mobile teminal Mobile teminals: PDA, cell phone While using a client/seve achitectue. A PDA o a cell phone o a PC epesents the client Seve computes a ve small set of ke s Client utilizes these s to use a waping technique to compute new s as seen b intemediate cameas Intemediate cameas: positions and diections ae chosen inteactivel b the use b moving the stlus of a PDA. Most difficult poblem is how to place the cameas captuing the ke s Camea placement allows an efficient waping avoiding atifacts, such as holes, Holes due to occlusions and eposues. Poviding a geneal solution to the poblem of camea placement is a had task. We addessed onl the case of uban scenes.

2 Rendeing on Mobile Teminals Rendeing on Mobile Teminals Camea placement: a ve difficult task No satisfing solution available One Solution Fom a single econstuction of a coase 3D model Fom a set of s of a eal o vitual scene taken when walking though this scene Coase 3D model fo each Put in coespondence the obtained 3D models: use geomet waping Rendeing Fom a single Pupose ATIP A Tool fo 3D Navigation inside a Single Image with Automatic Camea Calibation Single Simple 3D navigation Kévin Boulange, Kadi Bouatouch, Sumanta Pattanaik IRISA, Univesité de Rennes I, Fance Univesit of Cental Floida, USA 2

3 Softwae achitectue Tou Into the Pictue ATIP Make : pocesses the input, ceates an.atip file ATIP Navigato : eads the.atip file, allows multi-platfom 3D navigation Output Oiginal method b Ho et al. Given an input, the pespective effect is detemined b manuall (mouse click) choosing a point in the as a vanishing point The scene is manuall appoimated b a bo b dagging its cone handles Tetues ae computed fo each bo s face fom the input The tetued bo is endeed fom anothe viewpoint Input.atip file ATIP ATIP make make ATIP ATIP navigato navigato bo with handles vanishing point 3 4 Tou Into the Pictue Automatic Tou Into the Pictue Manual fitting of the bo long and tedious Limitations suppots onl s with a single vanishing point hoizontal and vetical lines must be paallel to the bodes caeful captue of the input is necessa (sa: using a tipod) Contibutions of ATIP: minimal use inteaction moe accuate calibation of the camea using multiple vanishing points input fom an oientation of the camea is acceptable the camea can be hand-held 5 6

4 Vanishing points ATIP Make pipeline Vanishing points help to calibate the camea Alwas thee vanishing points in an Thee combinations of finite and infinite vanishing points TIP manages onl one finite vanishing point ATIP manages eve situation, even with otation aound the view diection (ve common without tipod) Hoizontal IVP FVP Vetical IVP Vetical IVP FVP FVP FVP FVP FVP Fist pat: pocessing Input Edge detection Dominant lines detection Vanishing points estimation Thee vanishing points 1 finite vanishing point, 2 infinite vanishing points Hoizon line 2 finite vanishing points, 1 infinite vanishing point 3 finite vanishing points 7 8 ATIP Make pipeline Edge detection Second pat: detemining paametes fo TIP (camea, bo, tetues) Convesion to gascale of the input Logical AND between gadient-based and laplacian-based methods Thee vanishing points Camea paamete etaction Bo size adjustment Tetue computation Tetues Geomet Input 9 Input Gascale Edges 10

5 Dominant lines detection Dominant lines detection 1-to-m Hough tansfom Dominant lines: longest alignments of points Vanishing lines = subset of the set of dominant lines How: 1-to-m Hough tansfom of the edge points high-pass filte of the Hough tansfom esult detection of the maima b thesholding tansfom the maima back to lines in space Line (with pola paametization) Point (intesection of an infinite numbe of lines) Image space ρ θ Hough space ρ ρ θ θ Point Cuve ρ =.cos(θ) +.sin(θ) Edges Hough tansfom High-pass filte Dominant lines 11 Aligned points ρ Set of cuves with a common point, θ coesponding to a set of aligned points in space 12 Dominant lines detection Hough space high-pass filte Vanishing points estimation Simple thesholding of the Hough tansfom lot of noise due to stongl tetued egions of the input High-pass filte to emove the dense egions of high values (noise) Region with points to be kept Dense egion of high values to be eliminated b filteing Hough tansfom Simple thesholding High-pass filte Final esult Kept points Noise eliminated 13 Vanishing point = intesection of a subset of dominant lines Pojection of each dominant line onto a hemisphee Accumulation of the esulting cuves on the hemisphee Retieve the thee maima Poject them back to space to get the vanishing point coodinates Pojection of the 2D line Image plane θ φ z Line in space to poject 14

6 Vanishing points estimation Accumulation Vanishing points estimation Thee maima etieval The intesection of cuves on the hemisphee coesponds to the intesection of the coesponding lines in space Lines to poject z Image plane Final intesection point Intesection point of the cicles Pojection of one line (pole) (equato) 0 0 C z 0 Gid of accumulatos )) i Hoizontal infinite vanishing point θ φ Finite vanishing point (close to the cente) (,,z) N/4 N/2 3N/4 N-1 Vetical infinite vanishing point Stips of equal aea Non-unifom subdivision of the hemisphee i 15 The thee maima cannot be etieved in a single step, instead : fo i = 0 to 2 { filteedcells = low_pass_filte(hemisphee_cells); macell = maimum_value_cell(filteedcells); vanpoints[i] = poject_to plane(macell); dominantlines[i] = dominant_lines_associated_with(vanpoints[i]); hemisphee_cells = negative_accumulation(dominantlines[i], hemisphee_cells) } Hpothesis: the thee esulting vanishing points coespond to othogonal diections With this algoithm, onl dominant lines that contibute to the detection of vanishing points ae consideed 16 Camea paamete etaction Fom the thee vanishing points, finite o infinite, etact camea paametes Focal length and otation mati The thee vanishing point diections define the wold coodinate fame Pocess the thee combinations of vanishing points diffeent was Camea paamete etaction One finite VP, two infinite VPs V 1 finite vanishing point I 1 and I 2 diections of the infinite vanishing points in space O position of the camea,, z coodinate fame of the camea (fied) (focal length) is fied b the use (48 b default) u f, v, w wold coodinate fame (columns of the otation mati), to be found I I I, 0 I OV = V V, f I I, 0 plane I 1 I P 2 f V 1 v u O w 17 z 1 ( ) T 1, 1 1 = I1V1 + I u = I1, I1, f ( ) T 1, 1 I 2V1 + I v = I 2, I 2, f w = V V, f ( ) T 1, 1 T 1V1 T 2 V1 2 = u = v = w = ( ) T, 2 2 u u v v w w 18

7 Camea paamete etaction Camea paamete etaction Thee finite vanishing points, v1, v 2 and v3 Æ oveconstained poblem ( v1, v 2 ), The two finite VPs method used fo (v 2, v 3 ) and (v 3, v1 ) The dot poduct between the thid VP diection and the coss poduct of the two fist diections gives an eo value Choose the set of vectos giving minimal eo Two finite VPs, one infinite VP Thee finite vanishing points Two known VPs ( V1 and V2 ), epesent othogonal diections f can be computed v obtained b coss poduct OV1 = (V1,V1, f ) T OV2 = (V2, V2, f ) T plane V2 OV1.OV2 = 0 V1 P f >0 f = V1V1 + V2 V2 f v O u w z OV1 u= OV1 OV2 w= OV2 v = w u 19 Bo size adjustment 20 Tetue computation The onl step of the algoithm that equies use inteaction Simple than TIP since the camea is calibated, onl the cones (ed handles) have to be moved Data etieved fom the input Each bo s face is subdivided into a unifom gid, each cell coesponds to a teel Each teel cente point of a tetue is pojected onto the input plane Bilinea intepolation gives the final colo Points falling outside the input bounds ae set to black tetued quadilateal plane (,,z) (i,i) O 21 22

8 Results Results Results with difficult scenes Fom multiple s Foest scene: Robust dominant lines detection fom nois edges using high-pass filte on Hough tansfom Non-flat sufaces: Robust «maima on the hemisphee» computation algoithm Hand-dawn sketch: Coect detection of dominant lines fom nois edges and impecise intesections of vanishing lines 25 Solution Choose one camea fame CF Detemine the tansfomations between CF and anothe camea fame ACF Tansfom the 3D model, found fo one ACF, to CF Mege the two models, esult = combined model CM CM epessed in CF fame + CM epessed in ACF fame Fo an intemediate camea, intepolate between each pai of coesponding vetices: geomet and tetues 20

9 ATIP with multiple s Questions Solution Seve: sends the client the combined 3D models associated with two o moe successive s captued b the camea fo diffeent positions and oientations One 3D model = N faces + N tetues Client: endes the eceived 3D models as well as inbetween intepolated 3D models 20 Kévin Boulange, Kadi Bouatouch, Sumanta Pattanaik. ATIP: A Tool fo 3D Navigation inside a Single Image with Automatic Camea Calibation. EGUK 2006, june 2006 Gwenola Thomas, Géald Point, Kadi Bouatouch A Client-Seve Appoach to Image-Based Rendeing on Mobile Teminals Technical Repot RR-5447, INRIA Rendeing on Mobile Teminals A Client-Seve Appoach to IBR Rendeing on Mobile Teminals A Client-Seve Appoach to IBR The IBR pocess is initialized when the seve sends the client an initial efeence togethe with its coesponding camea paametes (1.Send_Init(IRef0)). IRef = a efeence and its coesponding camea paametes. On the client side, the use can navigate though the 3D envionment b changing the oientation and the position of the camea (2.Navigate(theta,d)). In the pesent application, navigation is pefomed in a uban envionment. The position of the camea is constained to lie on a hoizontal plane. two successive s to make the IBR appoach possible. These limitations ae also coheent with the wa people walk in a cit. Wheneve the use moves the navigation camea, the client computes a new b waping some of the available efeence s (2.1.Update_NavigationCamea(), 2.2.Poduce()). Available efeence s: not alwas appopiate fo waping, Available efeence s too fa fom the cuent navigation camea ma cause the appeaance of holes on the waped. We use bluing filttes to fill the appeaed holes. To maintain an appopiate set of efeence s on the client side, the client tansmits the paametes of the new cuent navigation camea to the seve wheneve the use moves the camea (2.3.Send_NavigationCamea(Mc)).

10 Rendeing on Mobile Teminals A Client-Seve Appoach to IBR Rendeing on Mobile Teminals A Client-Seve Appoach to IBR The set of efeence s, available on the client side, is appopiate if each efeence of this set significantl contibutes to the constuction of the waped. The contibution of a efeence is measued as the pecentage of piels of the efeence that e-poject onto the waped The seve owns the 3D uban scene and a set of edges that define the geomet of the steet netwok. Depending on the cuent navigation camea Mc on the client side and on the efeence s peviousl sent IRefi, the seve is able to detemine whethe the efeence s, available on the client side, have to be updated o not (2.4.update=Update_RefeenceImages({IRefi})). A efeence has to be eplaced on the client side when it does not signicantl contibute to the waped. If necessa updates, the seve sends the client new ef s (2.5.[update]Send_RefeenceImages({IRefi})). The wa the cameas ae positioned in the envionment and the wa the seve selects them to compute efeence s ae povided b the camea placement algoithm. Rendeing on Mobile Teminals A Client-Seve Appoach to IBR Rendeing on Mobile Teminals A Client-Seve Appoach to IBR The IBR pocess is initialized when the seve sends the client an initial efeence togethe with its coesponding camea paametes (1.Send_Init(IRef0)). IRef = a efeence and its coesponding camea paametes. On the client side, the use can navigate though the 3D envionment b changing the oientation and the position of the camea (2.Navigate(theta,d)). In the pesent application, navigation is pefomed in a uban envionment. The position of the camea is constained to lie on a hoizontal plane. two successive s to make the IBR appoach possible. These limitations ae also coheent with the wa people walk in a cit. Wheneve the use moves the navigation camea, the client computes a new b waping some of the available efeence s (2.1.Update_NavigationCamea(), 2.2.Poduce()). Available efeence s: not alwas appopiate fo waping, Available efeence s too fa fom the cuent navigation camea ma cause the appeaance of holes on the waped. We use bluing filttes to fill the appeaed holes. To maintain an appopiate set of efeence s on the client side, the client tansmits the paametes of the new cuent navigation camea to the seve wheneve the use moves the camea (2.3.Send_NavigationCamea(Mc)).

11 Rendeing on Mobile Teminals A Client-Seve Appoach to IBR Rendeing on Mobile Teminals A Client-Seve Appoach to IBR The set of efeence s, available on the client side, is appopiate if each efeence of this set significantl contibutes to the constuction of the waped. The contibution of a efeence is measued as the pecentage of piels of the efeence that e-poject onto the waped The seve owns the 3D uban scene and a set of edges that define the geomet of the steet netwok. Depending on the cuent navigation camea Mc on the client side and on the efeence s peviousl sent IRefi, the seve is able to detemine whethe the efeence s, available on the client side, have to be updated o not (2.4.update=Update_RefeenceImages({IRefi})). A efeence has to be eplaced on the client side when it does not signicantl contibute to the waped. If necessa updates, the seve sends the client new ef s (2.5.[update]Send_RefeenceImages({IRefi})). The wa the cameas ae positioned in the envionment and the wa the seve selects them to compute efeence s ae povided b the camea placement algoithm.