A rate-distortion driven approach to remote visualization of 3D models

Transcription

1 A rate-storton rven approach to remote vsualzaton of 3D moels Petro Zanuttgh, Ncola Brusco, Guo Cortelazzo Unversty of Paova, Italy Dav Tauman UNSW, Australa Astract- Ths paper presents a novel approach for an nteractve 3D moels rowsng envronment. In the propose scheme oth texture an geometry ata s avalale at the server compresse n a scalale fashon an texture nformaton s represente y a set of vews of the 3D moel. The am s to show the est possle renerng of the current vew, contone on the avalalty of prevously transmtte nformaton for other neary vews, an suject to a transmsson uget constrant. The propose framewor ntrouces a rate-storton crtera to ece how to allocate transmsson resources etween the elvery of new texture ata an new elements from the geometry t-stream an how to len the avalale ata n orer to mnmze the storton n the renere vews. I. INTRODUCTION Remote vsualzaton of 3D moels s a new prolem posng many challengng conceptual an practcal questons. A asc ssue aresse n some prelmnary stues [1] s how to strute the avalale transmsson resources etween texture an geometry. What we present n ths chapter can e regare as a novel approach for the remote vsualzaton of 3D moels whch rests upon a rate storton theoretcal approach. It can also e regare as an nstrument for the expermental nvestgaton of how to suve geometry an texture nformaton wthn a preefne amount of ata n orer to acheve the est renerng results. The system s mae of a server an a clent connecte va a anlmte channel. The server stores the full moel, wth hgh qualty vews of t to e use as textures. At clent se, the user nteractvely etermnes the partcular vew of nterest. The server sens progressvely compresse 3D ata an texture, accorng to the user poston an avalale anwth. It sens only the part that, accorng to the server polcy, est fts the user s nees an the avalale anwth. The clent stores the 3D ata an texture t has receve, accorng to ts memory capacty. The avalale 3D ata an texture are comne accorng to the clent polcy an the renerng engne calculates the requeste vews. An overvew of the complete system archtecture s shown n fgure 1. Full 3D Moel Server Clent Polcy Server Polcy 3D Data an texture poston Mergng Data Partal 3D Moel Fgure 1. System structure Clent Renerng Wnow In ths paragm the clent oes not nee to now the server polcy. It tres to use the avalale ata the est way t can. The user can navgate the 3D envronment even f the connecton etween clent an server s own, snce the clent wll show only the ata t has. The system outlne aove gves rse to two funamental questons: How shoul the clent comne nformaton from avalale orgnal vew mages nto a new vew of nterest, usng the avalale geometry nformaton? How shoul the server strute avalale transmsson resources amongst the varous orgnal vew mages an the geometry nformaton? Ths paper s focuse on the frst queston, snce answers to the secon queston epen on how the server expects the clent to use the nformaton whch t has. Secton II escres how receve vews can e use to generate the requre renerng. Secton III evelops the rate-storton approach for the choce of source vews to e use n the renerng at the clent, whle Secton I proves some prelmnary expermental evence to valate ths approach. Secton outlnes some rectons whch we are currently nvestgatng.

2 II. GENERATING NOEL IEWS In ths secton the process of renerng a novel vew from a sngle orgnal vew mage s refly escre. We assume that the clent has the mage an a trangular mesh representaton for the surface geometry G, projectng ts noes onto the mage planes corresponng to an. Isometrc or perspectve projectons mght e employe, for example. Let n an n enote corresponng trangular patches of the two projecte meshes, as llustrate n Fg. 2. Of course some of the projecte trangles may e hen n one mage, ut not n the other. If a trangle s not vsle n, t s not use n the renerng, whle f t s mssng n t represents a hole that can t e flle from. To he these artfacts, an also to open the oor to resoluton-epenent sttchng algorthms, we prefer to form n the DWT (screte( wavelet ) transform) oman. Each possle renerng, W, s generate n the mage oman an separately sujecte to DWT analyss. Ths prouces a collecton of suans, LL D an LH, HL, LH for = 1, 2,..., D, where D s the numer of DWT ecomposton levels. Sttchng s carre out wthn the nvual suans to prouce LL D an LH, HL, LH, from whch s recovere y DWT synthess. A scheme of ths proceure s presente n fgure 3. Secton I proves vsle evence for the enefts of ths DWT-ase sttchng approach n * G * ( ) * * ( ) ( T ) Warpng DWT Synthess * ( T ) * ( ) DWT Analyss affne warpng T * A ( ( ( )) T ) ) Polcy Fgure 2. Renerng from an geometry G Each vsle trangle n s renere y affne warpng of n. We wrte ths as = W ( ) or, over the oman of each trangle, as n = W ( ) n. Amttely, affne warpng oes not exactly exten the ehavour of a perspectve magng moel nto the nteror of the projecte surface trangles. However, ths error can e renere artrarly small y reucng the sze of the surface mesh elements. In the propose scheme there are multple vews to e warpe at t s necessary to comne the fferent renerngs together. The smplest way to comne the nformaton from multple orgnal vew mages s to smply average the renerng otane y mappng each of them onto the esre vew. Unfortunately the mperfectons n the geometry escrpton prouce msalgnment amongst the separate renerngs an averagng tens to lur hgh frequency spatal features. An alternatve soluton s to rener each trangle from a sngle orgnal vew mage ( sttchng ) The followng sectons wll escre how to entfy the est source trangle. Ths approach avos the lurrng prolem ut prouces vsle scontnutes at the ounares etween ajacent trangles whch are renere from fferent orgnal source vews ue to geometry errors an lghtng ssues. Fgure 3. Sttchng n the DWT oman III. SELECTION OF THE BEST STITCHING SOURCE The renere vews at clent se show a certan error ue to varous types of storton. The strategy for selectng the est sttchng source for every trangle n s the mnmzaton of such an error, whch wll e formally expresse n ths secton. There are three sources of error. The frst, whch wll e calle quantzaton error for smplcty, s the qualty of mages that have een receve from the server. The secon error source s the warpng process, whch s oun to ntrouce some storton. Warpng can expan some areas n a relevant way an hgh frequences coul e mssng. The thr source of error s ue to geometrc moelng. A. Effects of quantzaton error The frst step s to escre a metho to select the est sttchng source, for each trangle n, ase solely on the amount of quantzaton error power. In the propose scheme mages are compresse n the DWT oman usng JPEG2000.

3 The quantzaton error assocate wth a sngle sample n suan of the mage, fns ts way nto suan of W ( ) through DWT synthess, warpng an further DWT analyss (Fg 4). (suans) S U A Dstorton * D ', D ', W ', T A( T ) Fgure 4. Propagaton of the quantzaton error If we call s the relevant suan synthess vector, an A the suan analyss operator whch prouces suan from an nput mage, then the square error n our orgnal suan sample s scale y W,n = A (T,n(s )) 2 to etermne ts contruton ( to the total square error n suan of W ). Here, T,n s the affne operator assocate wth W wthn trangle n. In the hypothess of orthogonal ass vectors, the total quantzaton storton appearng wthn the regon efne y n n suan ( of W ) can e approxmate ( gnorng the fact that the error sgnal s smeare out y the overlappng DWT ass vectors) y D,n, = W,n D,n, (1) where D,n, s the total square quantzaton error appearng wthn the regon efne y n n suan of the orgnal vew mage,. The est sttchng source coul potentally ffer from suan to suan, ut n the current system all samples corresponng to a sngle trangle are taen from the same vew to avo artfacts ue to msalgnment cause y geometrc errors. It s worth notng that the affne operator T,n stretches the synthess ass functon s y an amount n /, n amplfyng ts energy y roughly the same amount. Assumng an orthonormal transform, we can say that s = 1, T,n (s ) 2 = n / n an hence W,n = n /, n so that D,n, = n / n D,n, (2) The total storton n the warpe trangle seems to e roughly nepenent of the affne operator T,n, snce the total storton n the source trangle D,n,, shoul e roughly proportonal to ts area. Ths s not true ecause, frstly T,n must e a anlmte warpng operator, so that T,n (s ) 2 = n / n F (T,n (s )), where F (T,n (s )) s the fracton of the energy n T,n (s ) whch falls wthn the Nyqust samplng lmt. Secon, expansve operators T,n cannot prect the hghest frequency etals of at all. To ntrouce the effects of these two contrutons, the sums n (1) an (2) have een extene y nclung suans from a set of hypothetcal resolutons aove those of the orgnal mages. Snce there s no nformaton for these suans, ther stortons D,n, are consere equal to ther energes E,n,. The energy has een estmate y projectng each source mage onto the other n turn an tang the maxmum of the energy prouce y such projectons. The extra contruton ntrouce y mssng suans grows wth n / n, an s gger for source mages for whch T,n s most expansve, thus forcng the choce of mages n whch the requre trangle s gger. A rect applcaton of (1) woul requre the clent to now the values, D,n,, ut f the mage are compresse n JPEG2000 t s possle to otan a reasonale estmate of D,n, from the compresson parameters of the receve coe-locs. B. Effects of geometrc error If there were no errors on geometry the algorthm of the prevous secton woul suggest to choose the trangle from the vew n whch t s larger. Unfortunately the relalty of the 3D geometry epens on fferent causes. Frst of all, the accuracy of the acquston an 3D moelng tools (expensve tools le range-cameras can scan ojects wth hgh precson, whle passve 3D reconstructon methos can ntrouce larger errors). If the geometrc moel were hghly unrelale, we woul expect to otan etter results y selectng the orgnal vew mage whch s closer to the esre vew.. The uncertanty n the surface geometry translates nto uncertanty n the parameters of the affne transformatons, T,n. Ths ntrouces a translatonal uncertanty, whch has een stue prevously n [2]. Its effect may e moele y augmentng each term D,n, n (1) y a secon contruton of the form σg 2 φ2,n ω 2 E,n,. Here, ω s representatve of the spatal frequences elongng to suan, E,n, s the estmate suan energy efne aove, σg 2 reflects uncertanty (MSE) n the surface noe postons, an φ,n represents the senstvty of 2D mesh noe postons to splacements n the orgnal 3D surface noes. Ths moel oesn t tae nto account the llumnantepenent effects (le shang an reflectons). We can ntrouce an atonal term whch grows wth the evaton etween the orentaton of vews an. We expect ths storton term to e proportonal to the sgnal power, suggestng the followng augmente verson of equaton (1). D,n, = [ ] ) W,n (D,n, + σgφ 2 2,n ω 2 +g( n,n ) E,n, (3)

4 Here, n an n are the surface normals an, n the asence of careful moelng, g (x) s set to γ tan ( cos 1 x ), where γ etermnes the value we place on llumnaton felty. We expect the server to prove an ncatve value for σ 2 G. The propose scheme am to mnmze the storton locally n every sngle trangle. Though locally optmal, tang close trangles from many fferent vews coul lea to vsual artfacts n the fnal renere mage. These can e avoe y ntroucng a regularzaton algorthm that forces close trangles to e taen from the same vew. The actual system checs for every vertex of every trangle the mesh elements that elong to t. If more than half of them (nclung the trangle tself) are taen from the same vew, the current choce for that trangle s upate. The step can e repeate several tmes, however the algorthm converges qucly. I. EXPERIMENTAL RESULTS In ths secton we present some expermental results aout the performance of the system. In our expermental settng we conser four orgnal mages, = 0,..., 3 of an oject (Santa Claus), taen 90 egrees apart whch have een sent to the clent together wth geometry nformaton G represente y a 1000 trangles mesh ult usng passve methos. Fgure 5 shows the result of sttchng n the mage oman, whle the other pctures of fgure 5 show the results of sttchng n the DWT oman accorng to polcy functon ase on storton expressons from equaton 1 (Fgure 5c an 5) an 3 (Fgure 5e an 5f) respectvely. Images 5 an 5f clearly show that the complete storton moel leas to a etter mage qualty. Fgure 6 refers to a reconstructon from 2 mages an a 5000 trangles mesh of a moel of a frog. The frst mage (ar green) s very close to the esre vewpont, whle the secon one (lght green) s farther. Images 6c an 6 show how tang nto account the components of the storton ue to geometrc moelng errors an lghtng (eq. 3) forces the choce of trangles from the closer vewpont. Fnally Fgure 7 shows how the trangles choce s mofe when one of the two mages has een transmtte wth a goo qualty (shown n re), whle for the other only some pacets of ata has een receve (shown n green, lower qualty). As expecte less trangles are taen from the green mage when ts qualty s reuce. (c) (e) Fgure 5. Santa Claus from mages wth the same qualty a) a photograph taen from the vewng poston ) renerng wth sttchng wthout DWT; c) sttchng source an ) sttchng ase on DWT an storton (equaton (1)); e) sttchng source an f) sttchng ase on DWT an storton (equaton (3)); () (f). FURTHER RESEARCH Some ulng locs of the propose framewor are stll mssng. Many solutons for scalale cong of the surface mesh have alreay een propose such as [3],[4],[5].

5 tell us whch mage the clent wll select as ts sttchng source for each trangle, entfyng the mpact of texture storton D,n, an geometry storton σg 2 on the qualty of the reconstructe vew. Servce polces themselves, however, are eyon the scope of ths present paper. (c) Fgure 6. Frog from two mages a,) wth sttchng ase on storton; c,) wth sttchng ase on storton an geometrc error () I. CONCLUSIONS In ths paper we propose a new approach to the remote vsualzaton of 3D moels that tres to offer a traeoff etween the classc framewor of three mensonal computer graphcs an Image-Base Renerng technques. A partcular attenton s ecate to the optmal struton of compresse ata etween mages an geometry nformaton. The propose system offers a reasonale estmate of the storton assocate wth renerng an ntene vew from a varety of compresse mages wth uncertan geometry. The correctness of the propose storton moel has een confrme y expermental results. ACKNOWLEDGMENT Ths wor was supporte y FIRB PRIMO. REFERENCES [1] I. Cheng an A. Basu, Relalty an jugng fatgue reucton n 3 perceptual qualty, n The 3r Internatonal Symposum on 3DPT, IEEE, Septemer [2] D. Tauman an A. Secer, Hghly scalale veo compresson wth scalale moton cong, Proceengs of Internatonal Conference on Image Processng, vol. 3, pp v.2, Septemer Fgure 7. Choce of trangles from two mages a) mages wth the same qualty; ) green mage wth lower qualty JPEG2000 represents an excellent choce for the scalale compresson of each. It s also possle to nclue the recent JPIP stanar [6] to acheve an effcent ssemnaton of ncremental contrutons from the scalale compresse t-streams, uner server control, whch can e extene to ncorporate the elements from scalale geometry moels. The ey mssng element s a rate-storton ase framewor at the server to ece whch n of ata, new vews, ncremental nformaton for the avalale vews or geometry nformaton, shoul e transmtte n orer to otan the est renerng output. Ths framewor can e erve startng from the clent polcy of equatons 1 an 3. In partcular, these equatons [3] M. Garlan an P. Hecert, Surface smplfcaton usng quarc error metrcs, [4] H. Hoppe, Progressve meshes, Computer Graphcs, vol. 30, no. Annual Conference Seres, pp , [5] I. Gusov, W. Swelens, an P. Schröer, Multresoluton sgnal processng for meshes, Computer Graphcs Proceengs (SIGGRAPH 99), pp , [6] D.Tauman an R.Pranoln, Archtecture, phlosophy an performance of jpp: nternet protocol stanar for JPEG 2000, n Int. Symp. sual Comm. an Image Proc., vol. 5150, pp , IEEE, July 2003.