3-D IMAGE MODELS AND COMPRESSION - SYNTHETIC HYBRID OR NATURAL FIT?

Transcription

1 3-D IMAGE MODELS AND COMPRESSION - SYNTHETIC HYBRID OR NATURAL FIT? Bernd Girod, Peter Eisert, Marus Magnor, Ekehard Steinbah, Thomas Wiegand Te {girod eommuniations Laboratory, University of Erlangen-Nuremberg Cauerstrasse 7, Erlangen, Germany eisertlmagnorlsteinbiwieg~d}~nt.e-tehnik.uni-erl~gen.de Invited Paper ABSTRACT This paper highlights reent advanes in image ompression aided by 3-D geometry information. As two examples, we present a model-aided video oder for effiient ompression of head-and-shoulder senes and a geometry-aided oder for 4-D light fields for image-based rendering. Both examp les illustrate that an expliit representation of 3-D geometry is advintageous if many views of the same 3-D objet or sene have to be enoded. Waveform-oding and 3-D model-based oding an be ombined in a rate-distortion framework, suh that the generality of waveform oding and the effiieny of 3-D models are available where needed. 1. INTRODUCTION Soure models play an important role in image and video oding. Knowledge that is available a priori and that an be represented appropriately need not be transmitted. Rate distortion theory allows us to alulate a lower bound for the average bit-rate of any oder, if a maximum permissible average distortion may not be exeeded. Often, pratial shemes perform lose to their rate-distortion theoretial bounds. It may not be onluded, however, that this fundamentally prevents us from inventing even more effiient oding shemes. Rate distortion theoretial bounds are valid only for a given soure model, and often these models are rather rude. A more sophistiated soure model might result in a lower rate at a given distortion. Better soure models are the key to more effiient image ompression shemes. The majority of images are the result of a amera pointing to a three-dimensional sene. The sene onsists mostly of surfaes refleting the illumination towards the amera aording to well understood physial laws. Threedimensional models throughout this paper are models ap turing the three-dimensional spatial struture of a sene in front of the amera along with the optial and photometri laws that govern the image formation proess. Given that 3-D models seem suh a natural fit for image ompression, their suess for this appliation has been remarkably poor. Almost all pratial ompression shemes are based on random proess models that ignore the 3-D nature of the world being imaged. The attempt to expliitly reover 3-D struture for a still image and use this information for oding is not very promising. The projetion of the 3-D sene onto the image plane is an enormous data redution, and a 3-D~reonstrution has to overome many ambiguities. How, for example, would one enode a (flat) photograph in a 3-D sene? We may, however, benefit from an expliit 3-D model when enoding a large set of'2-d images, where eah individual image represents essentially the same 3-D sene, but possibly from a different viewing angle and/or at a different point in time. For example, for a video sequene resulting from a amera moving through a stati 3-D (Lambertian) environment, we would ideally transmit a texture-mapped 3-D model of the environment one, and then only update the 3-D motion parameters of the amera. In this paper, we show two examples of how expliit 3-D models an improve image ompression. The first example, presented in Setion 2, is a lassi: model-based ompression of head-and-shoulder views for videotelephony. The seond example-(setion 3) is an area of reently inreased interest: ompression of light fields. An earlier version of this paper has appeared in [l]. Relative to [l], we have inluded new results in Setion MODEL-AIDED COMPRESSION OF VIDEOPHONE SEQUENCES For videotelephony, we want to transmit the head-andshoulder view of a talking person. More than 15 years ago, Forhheimer et al. have proposed a videotelephone system based on a omputer-animated 3-D head model [2] [3], and many groups have investigated suh systems sine [4]. Impressive progress has been made in the automati traking of faial expressions over the last few years [5]. For head-and-shoulder senes, bit-rates of about 1 kbps with aeptable quality an be ahieved. Unfortunately, a major drawbak of suh a system is still its limitation to a speifi 3-D model and hene lak of generality. In the following, we desribe an extension of an H.263 video ode [6] that utilizes information from a model-based ode. Instead of exlusively prediting the urrent frame of the video sequene from the previous deoded frame, predition from the syntheti frame of the model-based ode is additionally allowed. The enoder deides whih predi /99/ $ IEEE 525

2 Coder about I 1 kbltls Deoder Figure 1: Basi struture of the model-based ode. tion is more effiient in a rate-distortion sense. Hene, the oding effiieny does not derease below H.263 when the model-based ode annot desribe the urrent sene. On the other hand, if the objets in the sene orrespond to the 3-D models in the ode, a signifiant improvement in oding effiieny an be ahieved Model-based Video Code The struture of a model-based ode is depited in Fig. 1. The enoder analyzes the inoming frames and estimates the parameters of the 3-D motion and deformation of the head model. These deformations are represented by a set of faial animation parameters (FAPs) [7] that are entropyenoded and transmitted through the hannel. The 3-D head model and the faial expression synthesis are inorporated into the parameter estimation. The 3-D head model onsists of shape, texture, and the desription of faial expressions. For synthesis of faial expressions, the transmitted FAPs are used to deform the 3-D head model. Finally, individual video frames are approximated by simply rendering the 3-D head model. In our model-based oder all FAPs are estimated simultaneously using a hierarhial optial flow based method starting with an image of 88 x 72 pixels and ending with CIF resolution. In the optimization an analysis-synthesis loop is employed [8]. The mean squared error between the rendered head model and the urrent video frame is minimized by estimating hanges of the FAPs. To simplify the optimization in the high-dimensional parameter spae, a linearized solution is diretly omputed using information from the optial flow and motion onstraints from the head model. This approximative solution is used to ompensate the differenes between the video frame and the orresponding syntheti model frame. The remaining linearization errors are redued by repeating the proedure at different levels of resolution. For more details about the model-based ode please refer to [5] Proposed General Video Code Fig. 2 shows the arhiteture of the general, model-aided video ode (MAC). This figure depits the well-known hybrid video oding loop that is extended by a model-based ode. The model-based ode is running simultaneously to the hybrid video ode, generating a syntheti model frame. This model frame is employed as a seond referene frame for blok-based motion ompensated predition (MCP) in addition to the previous reonstruted referene frame. For eah blok the video oder deides whih of the two frames to use for MCP. The bit-rate redution for the proposed sheme arises from those parts in the image that are well approximated by the model frame. For these bloks, the bit-rate required for transmission of the motion vetor and DCT oeffiients for the residual oding is often highly redued. For more details about the arhiteture and the mode deision, see [9] in these proeedings Experimental Results Experiments are onduted for the standard CIF video test sequene Akiyo. The first 200 frames of this sequene are enoded at 10 Hz using both the H.263 and the model-aided H.263 oder. Sine no head shape information from a 3-D san is available for this sequene, a generi 3-D head model is used. Texture from the first video frame is mapped onto the objet. For omparison of the proposed oder with the anhor, the state-of-the-art test model of the H.263 standard (TMN-lo), rate-distortion urves are generated by varying the DCT quantizer parameter over the values 10,15,20,25, and 31. Bit-streams are generated that are deodable produing the same PSNR values as at the enoder. In our simulations, the data for the first intra-oded frame and the initial 3-D model are exluded from the results. This way we simulate steady-state behavior, i.e., we ompare the inter-frame oding performane of both odes exluding the transition phase at the beginning of the sequene. We first show rate-distortion urves for the proposed oder in omparison to the H.263 test model. The following abbreviations are used for the two odes: 526

3 frame residual aided oder. Both frames require about 720 bits. Signifiant visual improvements an be observed for the MAC ode. More experimental results an be found in [9]. multiframe moilon ompen- * sation - rnodelbased deoder 4 reonst. j frame j model : frame j motion vetors : FAPs modelbased oder Figure 2: Struture of the proposed model-aided video oder. Traditional blok-based MCP from the previous deoded frame is extended by predition from the urrent model frame. 0 TMN-10: The result produed by the H.263 test model, TMN-10, using Annexes D, F, I, J, and T. 0 MAC: Model-aided H.263 oder: H.263 extended by model-based predition with Annexes D, F, I, J, and T enabled as well Figure 4: Frame 150 of the Akiyo sequene oded at the same bit-rate using the TMN-10 and the MAC, upper image: TMN-10 (31.08 db PSNR, 720 bits), lower image: MAC (33.19 db PSNR, 725 bits). 3. LIGHT FIELD COMPRESSION U, Rate [kbivs] Figure 3: Rate-distortion plot for the video sequene Akiyo. Fig. 3 shows the results obtained for the test sequene Akiyo. Signifiant gains in oding effiieny are ahieved ompared to TMN-10. Bit-rate savings of about 35 % at equal average PSNR are ahieved at the low bit-rate end. The upper half of Fig. 4 shows frame 150 of the TMN- 10 oder, while the lower half orresponds to the model- Light Field Rendering (LFR) onstitutes a novel approah to generating arbitrary 2-D images of stati 3-D senes [lo, 111. Traditional 3-D rendering relies on geometry models, textures and lighting desriptions. In LFR, the sene s visual appearane from multiple viewing diretions, its light field, serves as basis for the rendering proess. A light field onsists of an array of onventional 2-D images. To attain photorealisti rendering results, light fields typially ontain several thousand images, making data ompression neessary for rendering, storing and transmitting light fields. Even though LFR does not depend on objet geometry, light-field oding an benefit from geometry information to ompensate disparity between images. Geometry has to be inferred from the light-field images, as light fields do 527

4 Figure 6: Approximate geometry model, reonstruted from the Airplane light-field images. Figure 5: The Airplane light-field onsists of a 2-D array of 8 x 8 images. not ontain expliit sene geometry. Disparity maps an be derived for aurate disparity ompensation between neighboring light-field images (see [la] in these proeedings), yet images farther away an only be disparity-ompensated at lower image resolution. If the light-field sene exhibits texture or silhouette information, approximate 3-D objet geometry an be reonstruted. 3-D geometry aids in oding of light fields by enabling disparity ompensation of arbitrarily many light-field images over any distane at onstant geometry oding bit-rate. Light-field oding with geometry information is demonstrated using the Airplane light field shown in Fig. 5. First, an approximate geometry model is derived from the 8 x 8 light-field images using the reonstrution algorithm desribed in [13] (Fig. 6). To ode the approximate geometry model with adjustable auray, the Embedded Mesh Coding (EMC) algorithm desribed in [14] is employed to the reonstruted Airplane geometry. Fig. 7 shows the model oded at different resolutions. The geometry is used to generate high-resolution disparity maps for all images, and the light field is hierarhially oded as desribed in [12]. In Fig. 8, geometry-aided oding performane is ompared with results from the disparity-map oder desribed in [12] and a blok-based light-field oder [15]. Bit-rate for oding the geometry model is negleted to illustrate the possible oding gain for light fields onsisting of several thousand images: full 3-D geometry yields up to 25% better ompression over blok disparity maps, and up to 40% lower bit-rate is ahieved ompared to blok-based light-field oding. If oding the geometry model is taken into aount, oding gain from expliit 3-D geometry ompared to [la] is minor for the Airplane light field beause of the low number of images to be oded. Fig. 9 depits oding performane for different geometry resolutions if geometry bit-rate is inluded. Note that this geometry bit-rate also inludes the bakside of the airplane that is never visible in Fig. 5. Depending on reonstrution quality, optimum oding performane is ahieved with different resolution models. While low-resolution geometry models do not ompensate disparity well and might even introdue additional error, too detailed geometry degrades oding performane due to the inreased geometry oding bit-rate. Optimum bit-rate alloation between geometry and residual-error oding is attained by seleting model resolution depending on overall target bit-rate. 4. CONCLUSIONS We have onsidered two very different appliations in this paper: the model-based ompression of head-and-shoulder video sequenes and the ompression of 4-D light fields. Both appliations have in ommon that essentially the same 3-D objet is visible in many 2-D images, from different viewing angles or at different time instanes with deformation. We found that in both senarios, an expliit geometry model helps to redue the bit-rate. As the overhead 1127 triangles 3835 triangles bits bits 3.1% max. deviation 0.4% max. deviation Figure 7: Airplane geometry model, oded at different resolutions; maximum vertex displaement is measured relative to model size. 528

5 40 p [r 2 36 v) a.- E L Z 32 {f $301 1-,- 11 : 3 Ge;et; Mode; -1). Disparity Ma Coder 121 * - -0 Blok-basefoder Bit Rate [bpp] Figure 8: Rate-Distortion urves of different oders for the Airplane light field; geometry bit-rate is negleted [r 236- v) a z * % rel. deviation b % rel. deviation --A Disparity Map Coder [12] 281!l 0.b5 0:l 0.; : Bit Rate [bpp] Figure 9: Rate-Distortion urves for different model resolutions if geometry bit-rate is taken into aount. I for enoding of geometry information is distributed over a large number 2-D views, geometry-aided ompression beomes inreasingly attrative. An unresolved question is the minimum number of views, beyond whih geometry-aided enoding is superior. Our examples also illustrate that waveform-oding and 3-D model-based oding are not ompeting alternatives but should be ombined to support and omplement eah other. Both an be elegantly ombined in a rate-distortion framework, suh that the generality of waveform oding and the effiieny of 3-D models are available where needed. 5. REFERENCES [l] B. Girod, P. Eisert, M. Magnor, E. Steinbah, and T. Wiegand, 3-D imaging and ompression - syntheti hybrid or natural fit?, International Workshop on Syntheti-Natural Hybrid Coding and Three Dimensional Imaging IWSNHC3DI 99, Santorini, Greee, Sep [2] R. Forhheimer, 0. Fahlander, and T. Kronander, Low bit-rate oding through animation, Pro. International Piture Coding Symposium PCS 83, pp , Mar [3] R. Forhheimer, 0. Fahlander, and T. Kronander, A semanti approah to the transmission of fae images, Pro. International Piture Coding Symposium PCS 84, number 10.5, Jul [4] D. E. Pearson, Developments in model-based video oding, Proeedings of the IEEE, vol. 83, no. 6, pp , Ju~ [5] P. Eisert and B. Girod, Analyzing faial expressions for virtual onferening, IEEE Computer Graphis and Appliations, vol. 18, no. 5, pp , Sep [6] ITU-T Reommendation H.263 Version 2 (H.263+), Video Coding for Low Bitrate Communiation, Jan [7] ISO/IEC FDIS , Generi Coding of audiovisual objets: (MPEG-4 video), Final Draft International Standard, Doument N2502, [8] H. Li, P. Roivainen, and R. Forhheimer, 3-D motion estimation in model-based faial image oding, IEEE Transations on Pattern Analysis and Mahine Intelligene, vol. 15, no. 6, pp , Jun [9] P. Eisert, T. Wiegand, and B. Girod, Rate-distortioneffiient video ompression using a 3-D head model, Pro. International Conferene on Image Proessing ICIP 99, Kobe, Japan, Ot [lo] M. Levoy and P. Hanrahan, Light field rendering, SIGGRAPH 96 Conferene Proeedings, pp , Aug [ll] S. J. Gortler, R. Gneszzuk, R. Szeliski, and M. F. Cohen, The lumigraph, SIGGRAPH 96 Conferene Proeedings, pp , Aug [12] M. Magnor and B. Girod, Hierarhial oding of light fields with disparity maps, Pro. International Conferene on Image Proessing ICIP-99, Kobe, Japan, Ot [13] P. Eisert, E. Steinbah, and B. Girod, Multihypothesis volumetri reonstrution of 3-D objets from multiple alibrated amera views, Pro. International Conferene on Aoustis, Speeh, and Signal Proessing ICASSP 99, Phoenix, USA, pp , Mar [14] M. Magnor and B. Girod, Fully embedded oding of triangle meshes, Pro. Vision, Modeling, and Visualization VMV 99, Erlangen, Germany, Nov [15] M. Magnor and B. Girod, Adaptive blok-based light field oding, Pro. International Workshop on Syntheti-Natural Hybrid Coding and Three Dimensional Imaging IWSNHC3D1 99, Santorini, Greee, Sept