Improving Quality of Free-Viewpoint Image by Mesh Based 3D Shape Deformation

Transcription

1 Imrovg Qualty of Free-Vewot Image by Mesh Based 3D Shae Deformato Satosh Yaguch Hdeo Sato Deartmet of Iformato ad Comuter Scece, Keo Uversty 3-4- Hyosh, Kohoku-ku, Yokohama, 3-85 JAPAN NTT COMWARE Cororato, NTT Shagawa TWINS Aex Bldg. E-mal -9- Koa, Mato-ku, Tokyo 8-89 JAPAN ABSTRACT I ths aer, we reset a method to sythesze hgh-qualty vrtual vewot mage targetg the detaled texture objects. About 3 mages are take from multle ucalbrated cameras aroud the object, ad the Vsual Hull model s recostructed wth Shae from Slhouette method. To deform 3D surface model that s coverted from Vsual Hull Model usg the formato such as mage texture ad object slhouette, the dfferece betwee the real object ad the recostructed model s evaluated as a cost fucto of otmzato roblem. Our deformg model algorthm s based o sgle vertex teratve shftg. The vertex of surface tragle mesh s moved to the selected caddate ot that maxmzes the cost fucto. The cost fucto s cossted by four costrat crtera, texture correlato, smoothess, object slhouette, ad mesh shae regularty. I addto to the cost fucto, such as judgg mesh drecto ad combg / dvdg meshes are aled for refed 3D models to avod mesh foldg ad mesh sze ueveess. The refed model rovdes a qute accurate dese corresodg relatosh betwee the ut mages, so that hgh qualty mage ca be sytheszed at vrtual vewot. We also demostrate the roosed method by showg vrtual vewot mages to alyg the real mage that are take from multle ucalbrated cameras. Keywords Shae-from-Slhouette, the Vsual Hull, shae refemet, Image Based Rederg, weakly calbrated multle camera system. INTRODUCTION The acqusto of 3D geometrc formato ad geeratg vrtual vewot mages from multle cameras are studed may years ad stll researched actvely. The studes of those methods are categorzed to two basc methods, correlato-based stereo aroach ad the Vsual Hull [Lau94a] model based aroach. The Multle Basele Stereo method [Oku93a, Nay98a] eables acqusto of 3D geometrc Permsso to make dgtal or hard coes of all or art of ths work for ersoal or classroom use s grated wthout fee rovded that coes are ot made or dstrbuted for roft or commercal advatage ad that coes bear ths otce ad the full ctato o the frst age. To coy otherwse, or reublsh, to ost o servers or to redstrbute to lsts, requres ror secfc ermsso ad/or a fee. Joural of WSCG, ISSN 3-697, Vol.4, 6 Plze, Czech Reublc. Coyrght UNION Agecy Scece Press formato wth correlato based stereo method from multle cameras. The advatage of correlatobased aroaches s able to hadle cocave regos. I cotrast, t s ot stable to hadle occluded rego. Occluded rego causes losg xels o sytheszed vrtual vewot mages. Baseles betwee each camera are ot short eough to remove occluded rego, eve f there are several tes cameras [Sa3a]. O the other had, the Vsual Hull model based aroaches maly recostruct a 3D geometrc formato wth shae from slhouette method [Mata]. These aroaches ca sythesze vrtual vewot mage wthout losg xels eve artally occluded regos. However, the dfferece betwee the real object ad recostructed model may cause blurrg the sytheszed vrtual mages. It s maly caused by cocave rego or suffcet umber of cameras to curve a Vsual Hull model. Esecally to aly a object wth a detaled texture, the blur greatly effects mage qualty aearace. Therefore the qualty of the sytheszed vrtual Joural of WSCG 57 ISBN ISBN

2 vewot mage s degraded by accuracy of the recostructed shae. Recetly, the advaced aroaches that take to accout the both advatages are tesvely studed. Those aroaches mrove the mage qualty to refe 3D geometrc shae wth varous formato such as a texture correlato. The Sace Curvg Method [Kuta, Slaa] removes uecessary voxels of the voxel-rereseted model usg texture formato by reducg dfferece betwee costructed models. The techques for otmzg 3D model by deformg the vertex of surface tragle mesh based o the correlato of the texture have also bee roosed [Eck4a, Nob3a]. We have bee studed acqusto of 3D geometrc formato ad geeratg vrtual vewot mages from multle cameras. We have already roosed a ew framework for the Vsual Hull based vrtual vewot mage sytheszg method exadg to weakly calbrated multle cameras usg the "Projectve Grd Sace (PGS)"[Sa99a] whch the coordates are defed by eolar geometry betwee cameras stead of strog calbrato [Yaga, Yag4a]. I the framework, vrtual vewot mages ca be sytheszed wth Image Based Rederg usg corresodece ma derved from the model lke a vew morhg method [Se96a]. However, there s stll remaed the blurrg roblem, because the dfferece betwee the recostructed model ad the real object s ot well cosdered. The goal of ths aer s to mrove a qualty of vrtual vewot mage that s sytheszed wth the framework. The roosed method ths aer ams to reduce the blurrg effect to deform the recostructed model wth the formato such as mage texture or object slhouette. Qute accurate dese corresodece ma betwee the mages that s derved from the refed model eables sytheszg hgh qualty vrtual vewot mages wthout the blur.. PROPOSED METHOD The aroach of the roosed method s based o the "Projectve Grd Sace (PGS) framework [Sa99a], whch relates the 3D object sace wth D mage coordates. Usg PGS framework, recostructg 3D shae model wth Shae from Slhouette method ad geeratg vrtual vewot mages becomes ossble from ucalbrated multle cameras [Yag4a]. Iut mages are take by ucalbrated cameras that observe aroud a target object, ad slhouette mages are sytheszed from those mages. Several corresodece ots betwee each mage are extracted from feature ots, ad fudametal matrces, whch are used for relatg 3D object sace ad D mage lae wth the PGS framework, are calculated from those ots. (Please refer the aedx or those aers, the detals of the PGS framework s show.) A tal voxel model s recostructed by the shaefrom-slhouette method o the PGS. By alyg the Marchg Cubes algorthm, the Vsual Hull model s coverted to the surface reresetato model. The vertex of surface tragle mesh s moved to the selected caddate ot maxmzes the cost fucto that s cossted by four costrat crtera. The cost fucto ca be comuted oly o D mage doma accordg to the rojecto comutato from PGS to every mage. Obtag a dese corresodece ma betwee mages from the otmzed model, hgh qualty vrtual vew mages are sytheszed as the mage terolatg two ut mages. 3. SHAPE REFINEMENT Ths secto descrbes roosed refemet techque of the 3D-surface model by usg oly D mage doma formato. Model shae s refed by movg each vertex of surface tragle mesh deedetly. Each vertex of the surface model s vsted sequetally, ad the cost fucto s evaluated at tal osto of the vertex. If the evaluated value s uder threshold, caddate ots of refemet vertex are defed, ad each cost value s calculated at those caddate ots resectvely. The vertex s moved to the caddate ot that maxmzes the cost fucto. Whe all vertexes are vsted N tmes, or cost fucto s over threshold at all vertexes, shae refemet s fshed. 3. Vertex Posto Otmzato Our algorthm of deformg 3D shae s based o shftg a sgle vertex teratvely. Each vertex of recostructed model s vsted resectvely ad shfted deedetly. The rocess of shftg vertex osto s erformed by selectg ad movg the caddate vertex to the ot that maxmzes cost fucto. The caddate vertexes are defed every teratg cycle. To smlfy the algorthm, the caddate ots are defed o the le assg through the target vertex v ad a ot g that s defed as the ceter of all adjacet vertexes x, x,, x m. The + caddate ots v,, v, (wth the target vertex) are defed outsde ad sde of the model surface at tervals of a ut vector scaled by a weght s (see Fgure ). Joural of WSCG 58 ISBN ISBN

3 The arameter s s decded dyamcally deedg o the value of the cost fucto of v to eable detaled search, ad that s raged. s.. The umber of caddate ot s decded deedg o the dstace betwee v ad g to reserve tal shae detaled structures. Fgure. Deform of vertex: Movg caddate ots are defed o a le v to g. Cost fucto V, V +, V,, V of each caddate ot v s calculated by the followg equato (). V = α V corr ( v ) + β V + λ V + δ V smooth shae sl ( v ( v ( v V corr : Texture correlato V : Smoothess smooth shae ) ) ) () V : Tragular shae regularty V sl : Slhouette where α, β, γ, δ, ε are weghtg coeffcets. Defto of each crtero s descrbed ext. 3.. Texture Correlato Texture correlato of a vertex s determed by the texture of ts adjacet tragle meshes. The correlatos betwee each mage are calculated by ormalzed cross correlato to aly all xels of the mesh. The correlatos are defed for the ut mages from whch the vertex ad the adjacet meshes are able to observe. Vertexes of tragle mesh v, x, x the mage, ad v j, x j, x j the mage j are related to 3D osto by fudametal matrces. Ier ot of tragle mesh the mage s corresoded to the ot the mage j by affe trasform A j (see Fgure ). Fgure. Texture correlato: Adjacet tragle Meshes are rojected o mages by fudametal matrxes. Each xel of mesh s corresoded other mages by Affe trasformato matrxes. The correlato of v of adjacet mesh k betwee mage ad mage j s show equato (). V corr ( v ) = ( W ( W w )( W w ) ( W kj j j j w ) j w ) () where W ad W j are the color of the ot ad j, resectvely, w, w j are the averages of color of all xel the mage ad j, resectvely. V corr ( v ) s also calculated for all combato of k j ars of mages o whch the vertex v s rojected oto the ar mages wthout occluso. The total V corr ( v ) s comuted as the average of all V corr kj calculated for the vertex v. V corr ( v ) s raged k j < V ( v ) <. corr 3.. Smoothess Costrat Though surface of the object should be locally smooth, ad be cotuous, we aly to followg smoothess costrat. The costrat s defed deedg o the dstace d ( v ) betwee the vertex v ad g, whch s determed by all adjacet vertexes of the target vertex as descrbed 3.. I order to reduce over-smoothg, the dstace d( v ) betwee g ad the ot v, whch s the /6 dstace ot o the le segmet betwee v ad g, s subtracted from d ( v ). Thus, we aly to followg fucto as the smoothess costrat. V smooth ( d( v ) d( ) / ) ( v ) = v (4) 6 where d s the dstace betwee v ad g, ad weghtg coeffcet β wll be egatve. Joural of WSCG 59 ISBN ISBN

4 3..3 Slhouette Costrat The 3D-surface model after deformed should be flled over the tal Vsual Hull suffcetly. The vertexes that determe the Vsual Hull are costraed to the boudary of the tal slhouettes. However, because the Vsual Hull model does't accurately exress the real cotour, for stace, the cocave regos, the vertex rojected o the boudary of the slhouette s oly a caddate determg the cotour. The slhouette costrat kees the refed model to form the tal slhouette. Therefore the followg slhouette costrat s aled to the target vertex ad the refemet caddate ots (see Fgure3). After all, V sl ( v ) of the caddate ot s defed by the sum of V v ) as followg equato (5). sl sl ( V ( v ) ) d (5) = Vsl ( v = 3..4 Costrat o Tragular shae regularty As teratve vertex deformg rocess goes o, mesh shae ad sze are chaged. If tragle of mesh s very small or very arrow, ts texture s ot able to derve eough to calculate correlato. So t s referable that tragle shae s ket shae regularty ad sze as same as ossble. Therefore, the crtero of costrats o the tragular shae regularty V shae ( v ) of adjacet mesh k of vertex v s show by equato (6) [Eck4a]. r r e e Vshae k ( v ) = 3 r r r, e + e + e 3 < V shae (6) where e r, e r, e r 3 are edge vectors. Ths fucto reresets the geometrc qualty of tragle area ratos The crtero V shae ( v ) of vertex v s defed the average of V v ). shae k ( Fgure 3. Slhouette Costrat: A mage that vertex rojected o edge, slhouette costrat crtero s defed deedg o the dstace from edge.. Whe the target vertex v s ot rojected oto the boudary of the slhouette ut mage, V s ot defed ( V v ) of all caddate sl ots are decded to.). sl (. Whe v s rojected oto the boudary of the slhouette o ut mage, Vsl s defed roorto to the dstace betwee the rojected coordate of caddate ots v ad that of v. I addto, 3. If v s rojected o outer slhouette eve by oe ut mage, v s excluded from caddate. 3. Addtoal Costrats Gog o the teratg rocess, vertex shft may cause mesh foldg or mesh sze ueveess. Mesh foldg chages the toology of the model or the vsblty of the vertex, the correlato of the mesh s ot able to calculate accurately ad the mesh s ot redered correctly (such as hole or overla texture s aeared the sytheszed mage). Mesh sze varato causes equalty of the recso of cost fucto, ad very small meshes cause valdate vertex because of havg ot eough areas dervg textures. I order to avod such foldg or ueveess of the mesh sze, followg addtoal costrats are also cosdered. 3.. Avodg Mesh Foldg Mesh foldg s occurred whe the mesh s ot occluded turs to be occluded. It s stated dfferetly, t s caused whe the vertex s moved over the boudary of the adjacet mesh. To avod such mesh foldg, ormal drecto of mesh o the rojected mage ca be used. Vertexes Joural of WSCG 6 ISBN ISBN

5 of mesh v, x, x are dexed the order of clockwse o the rojected mage, f the mesh s ot occluded, as show Fgure 4(a). The the drecto of surface ormal of the all the meshes adjacet to v are calculated wth edge vector. Accordg to the drecto, every mesh s determed f t s o the rght sde or wrog sde. If oe of the meshes s ot o the rght sde, mesh s cosdered as foldg as show Fgure 4(b). Such caddate ots of v are excluded from the vertex searchg caddate ots that are defed EXPERIMENTAL RESULT The roosed method has bee tested wth several real objects. Iut mages are take by ucalbrated camera as color mages (64 48 xels BMP format). Results for two real objects, a Jaguar ad a elehat are show ths secto. 4. Jaguar The target object s a aer craft of "Jaguar" about cm cm cm. 36 mages were take aroud a target object wth a had-held camera as ut mages. Fgure 5. Real Images from that vrtual vew mage show Fgure 7 are sytheszed to terolate. Fgure 4. Mesh foldg: If the caddate ot s decded over the boudary of the adjacet mesh, rotato of vertexes ad surface ormal vector are chaged reverse. 3.. Mergg / Dvdg Meshes ad Vertexes The dstace betwee the vertexes s chaged after the teratg rocess. Costrat o tragular shae regularty descrbed 3..5 s ot able to avod mesh sze ueveess. So, whe every terato cycle s fshed, meshes ad vertexes are dvded or merged accordg to the dstace betwee adjacet vertexes. If the dstace betwee adjacet vertexes s over the threshold dstace D max, ew vertex s serted to the mddle ot these vertexes, ad tragles that sharg these vertexes are dvde to two tragles each. If the dstace betwee each adjacet vertex s less tha the threshold dstace D m, these vertexes are merged to oe vertex. Whe two vertexes should be merged, frst, the vertex after merged s decded for the mddle ot of these vertexes. Next, two tragles that share both vertexes are removed, ad the mesh dex refereces of each vertex are redexed to refer the ew vertex. Fgure 6. Ital surface model ad refed model terated N=3. Fgure 5 shows the examle of the ut mage. A tal surface model recostructed the PGS ad the refed model wth terato N=3 by roosed method are show Fgure 6. Vrtual vewot mage were sytheszed wth terolato rato 5:5 of two real mages as show Fgure 7. Here, Fgure () ad () rereset the mage sytheszed from tal model, ad the mage sytheszed from refed model terated N=3, resectvely. I the mage of tal model (Fgure 7 ()), the surface texture of the jaguar s blurred. The backgroud area mage (blue area) s also redered as the surface texture of the object. Those bad rederg effects are caused by accurate shae of the tal model. Esecally the whte ellse area, the dscotuous atter s see, that s because of wrog shae of the jaguar s foot. O the other had, such bad rederg effect s comletely removed refed model mage (Fgure 7 ()). Joural of WSCG 6 ISBN ISBN

6 Fgure 7. Vrtual vew mage (5)) wth real vewot mage, about the same qualty texture s acqured wth refed model. 4. Elehat The target object s a "elehat" about cm cm cm. 3 mages were take as ut mages. The examles of ut mages are show Fgure9. Fgure shows a tal surface model ad the refed model wth terato N=45 by roosed method. Fgure shows the sytheszed free vewot mages. Images o ether sde are referece real mages ad mddle mages are sytheszed chagg terolato-weghtg factor. Uer mages were sytheszed from tal model, ad lower were sytheszed from refed model terated wth N=45. I the uer mages, may backgroud mage areas (blur areas) are dstrbuted, ad the border of adjacet rectagular atches are blurred. O the other had, such bad rederg effects are reduced the mages redered wth the refed model. I ths way, free vewot mages wth mroved qualty texture as same as ut mages are able to sytheszed usg roosed method. Fgure 9. Examles of ut mages. Fgure 8. comarso of textures: () Rederg from tal 3D shae model. () Real mage texture about the same vewot as vrtual vewot. (3), (4), ad (5) Redered mages wth refed model of terato N=, N=, ad N=3, resectvely. Next, Fgure 8 shows the comarso of texture for each teratg umber, tal model, N=, N=, N=3, ad real vewot mage. It s uderstood that the mage qualty of sytheszed mage has bee mroved as the umber of teratg rocess s creased. The real mage show Fgure 8 () s take from about same vewot as vrtual vewot. By comarg N=3 mage (Fgure 8 Fgure. Ital surface model ad refed model terated N= CONCLUSION We roosed a method for mrovg the qualty of ew vew mage by deformg 3D surface model from ucalbrated multle cameras. The cost fucto of deformg 3D shae model s defed oly o D mage doma accordg to the rojecto comutato from PGS to every mage. Deformg 3D shae model wth the texture correlato ad other costrats reduces the dfferece betwee the recostructed model ad the real object. Joural of WSCG 6 ISBN ISBN

7 Fgure. Sytheszed free vewot mages. Images o ether sde are referece real mages, the mddle uer mages were sytheszed from tal model, ad lower were sytheszed from refed model terated wth N=45. Usg refed model removes the blur o the mage texture ad eables sytheszg hgh qualty free vewot mage. Our method requres oly about to 3 mages ad does ot requre strog calbrato, so that t s easy to use our method wth smle camera systems. For the future work, we wll exted ths roosed method to the dyamc evets ad sytheszed free vewots vdeo from ucalbrated smle camera systems. 6. REFERENCES [Eck4a] Eckert, G., Wgbermuhle, J. ad Nem, W. Mesh Based Shae Refemet for Recostructg 3D-Objects from Multle Images: The Frst Euroea Coferece o Vsual Meda Producto (CVMP4), Mar.4. [Che93a] Che, S. ad Wllams, L. Vew terolato for mage sythess: Proc. SIGGRAPH '93,.79-88, 993. [Cur96a] Curless, B. ad Levoy, M. A Volumetrc Method for Buldg Comlex Models from Rage Images: Proc.of SIGGRAPH '96, 996. [Kuta] Kutulakos, K.N. ad Setz, S.M. A Theory of Shae by Sace Carvg: IJCV(38), No.3,.99-8,. [Lau94a] Lauret, A. The Vsual Hul Cocet for Slhouette-based mage uderstadg: IEEE Tras Patter Adl. Mache Itell., 6(), 5-6, Feb [Nay98a] Narayaa, P.J., Rader, P.W. ad Kaade, T. Costructg Vrtual Worlds usg Dese Stereo: Proc. ICCV 98, 998. [Mata] Matusk, W., Buehler, C., Raskar, R., Gorlter, S. ad McMlla, L. Image-Based Vsual Hulls: Proc. of SIGGRAPH,. [Nob3a] Nobuhara, S. ad Matsuyama, T. Dyamc 3D Shae from Mult-Vewot Images usg Deformable Mesh Models: Proc. of 3rd Iteratoal Symosum o Image ad Sgal Processg ad Aalyss, Rome, Italy, Setember 8-,. 9-97, 3. [Oku93a] Okutom, M. ad Kaade, T. A Multle- Basele Stereo: IEEE Tras. o PAMI, Vol.5, No.4, , 993 [Sa99a] Sato, H. ad Kaade, T. Shae Recostructo Projectve Grd Sace from Large Number of Images: IEEE Proc. Comuter Vso ad Patter Recogto, Vol., , 999. [Sa3a] Sato, H., Baba, S., Kaade, T. Aearace- Based Vrtual Vew Geerato From Multcamera Vdeos Catured the 3-D Room: IEEE Tras. o Multmeda, vol.5, o.3, , Se. 3 [Se96a] Setz, S. M. ad Dyer, C. R. Vew Morhg: roc. of SIGGRAPH '96,. -3, 996. [Slaa] Slabaugh, G.G., Schafer, R.W. ad Has, M.C. Mult-Resoluto Sace Carvg Usg Level Set Methods: Proc.ICIP, Vol.II, ,. [Yaga] Yaguch, S. ad Sato, H. Arbtrary Vewot Vdeo Sythess from Ucalbrated Multle Cameras: WSCG' - the -th Iteratoal Coferece Cetral Euroe o Comuter Grahcs, Vsualzato ad Comuter Vso', Feb. Joural of WSCG 63 ISBN ISBN

8 [Yag4a] Yaguch, S. ad Sato, H. Arbtrary Vewot Vdeo Sythess from Multle Ucalbrated Cameras: IEEE Tras. o Systems, Ma ad Cyberetcs, PartB, vol. 34, o, PP , 4. APPENDIX I our method, 3D ot s related to D mage ot wthout estmatg the rojecto matrces by Projectve Grd Sace (PGS), whch ca be determed by oly fudametal matrces reresetg the eoler geometry betwee two bass cameras. Because the 3D coordate of PGS s deedetly defed by the camera mage coordates, 3D osto of ay samle ots does ot have to be measured. Projectve Grd Sace The Projectve Grd Sace (PGS) s defed by camera coordate of the two bass cameras. Each xel ot (, q) the frst bass camera mage defes oe grd le the sace. O the grd le, grd ode ots are defed by horzotal osto r the secod mage. Thus, the coordate P ad Q of PGS s decded by the horzotal coordate ad the vertcal coordate of the frst bass mage, ad the coordate R of PGS s decded by the horzotal coordate of the secod bass mage. Sce fudametal matrx F lmts the osto the secod bass vew o the eolar le l, r s suffcet for defg the grd ot. I ths way, the rojectve grd sace ca be defed by two bass vew mages, of whch ode ots are rereseted by (, q, r). 3D-D Mag As descrbed the revous secto, the PGS s defed by two bass vews, ad the ot the PGS s rereseted as A(, q, r). The ot A(,q,r) s rojected oto a (, q) ad a (r, Y) the frst bass mage ad the secod bass mage, resectvely. The ot a s rojected as the eolar le l o the secod bass vew exressed as equato (7). l = F q (7) where F reresets the fudametal matrx betwee the frst ad secod bass mages. The ot a s oto l, thus the coordate of a (r, Y) s decded (see Fgure ). Fgure. Projecto of 3D ot oto a mage: The ot o the rojectve grd sace s rojected to the cross ot of two eolar les the mage of vew. The rojected ot th arbtrary real mage s determed two fudametal matrces, F, F betwee two bass mages ad th mage. The rojected ot the th mage must be o the eolar le l of a the frst bass mage, whch s derved by the F. I the same way, the rojected ot the th mage must be o the eolar le l of a, whch s derved by the F.The tersecto ot betwee the eolar le l ad l s the rojected ot A(, q, r) the th mage. I ths way, every ot of the PGS are rojected oto every mage, where the relatosh ca be rereseted by oly the fudametal matrces betwee the mage ad two bass mages. I ths way the PGS, the fudametal matrces betwee mages determes 3D-D mag. I the case of recostructg 3D shae model wth the shae-from-slhouette method, the voxels wth a certa rego of the PGS are rojected to the slhouette mage. Ad the ot a mage s able to corresod to other mage ots to use corresodece ma derved from the recostructed model. Ad so the corresodece ma eables to sythesze ew vew mages as terolated vews ad eables to calculate texture correlato betwee ut mages. Joural of WSCG 64 ISBN ISBN