Graph-Based vs Depth-Based Data Representation for Multiview Images

Transcription

1 Graph-Based vs Depth-Based Data Representation for Multiview Images Thomas Maugey, Antonio Ortega, Pasal Frossard Signal Proessing Laboratory (LTS), Eole Polytehnique Fédérale de Lausanne (EPFL) Department of Eletrial Engineering, University of Southern California Abstrat In this paper, we propose a representation and oding method for multiview images. As an alternative to depthbased shemes, we propose a representation that aptures the geometry and the dependenies between pixels in different views in the form of onnetions in a graph. In our approah it is possible to perform ompression of the geometry information and to preserve a diret ontrol of the effet of geometry approximation on view reonstrution. This is not possible with lassial depth-based representations. As a results, our method leads to more aurate view predition, when ompared to onventional lossy oding of depth maps operating at the same bit rate. We finally show in experiments that our representation adapts the amount of transmitted geometry to the omplexity of the preditions that are performed at the deoder. I. INTRODUCTION Multiview or D data are generally represented as a set of images that orrespond to the information aptured by ameras at different viewpoints. These images desribe the olor information aquired by the multiple ameras, along with depth information that beomes easily aessible []. Depth images desribe the distane between the sene and the foal length of the amera. Obviously this brings interesting hallenges for D transmission systems. For example, a depth image an be used to projet one referene image onto another one [], [] with interesting benefits in the ompression of multiview images. Despite the huge potential of this tool, one of the important questions linked with depth images relies in the effet of ompression on the view predition performane []. More preisely, an impreise depth value leads to a spatial position unertainty for the projeted pixels in the predited viewpoint. The modeling of this error has led to several works [], [6]. Some sophistiated depth image oder has also been proposed [7], [8] to takle this drawbak. However, artifats due to ompressed depth images remain generally diffiult to ontrol. This is why, in this paper, we propose a new multiview image representation that permits better ontrol of the geometry information. We propose a natural form of geometry information that is of moderate size, but leads to effetive view reonstrution algorithms. After observing that the knowledge of the sene geometry leads to onnetions between pixels in images from different viewpoints, we propose to diretly represent these links with a graph. The graph ontains all the geometrial information needed for multiview image reonstrution at the deoder side. Contrary to depth maps, the geometry information in our graph takes into aount the omplexity of the predition when adjusting the proper amount of geometry to ompress and transmit. The advantage of suh an approah is that it works diretly with the inter-pixel onnetions and offers a better ontrol of the geometry ompression artifats. We have ompared this approah with depth map representations in terms of view predition quality for similar geometry rate budget and shown promising with performane improvements of db. The paper is organized as follows. In Setion II, we introdue the general onepts of our new representation along with the main differenes with depth-based approahes. In Setion III, we explain the onstrution of the graph in detail and finally, in Setion IV, we validate the potential of our solution in terms of predition quality improvement. II. DENSE DISPARITY MAPS As mentioned in the introdution, one of the most adopted format to represent and transmit N viewpoints is the MVD one. It onsists of N olor image and their assoiated N depth maps. A depth map is a gray-sale D image that represents the distane between the sene and the amera plane. Sine it takes values between 0 and as a lassial image, it is assoiated to a saling funtion that onverts the hrominane values into units denoting the distane to objets. This saling funtion is generally not linear but follows a evolution in z in order to desribe more finely the losest points than the furthest objets in the sene. It is justified by the fat that the disparity of a point in the sene between two images evolves in funtion of the inverse of the depth. Depth images are mainly used to projet the orresponding olor image onto other arbitrary viewpoints. This is generally done with depth image based rendering (DIBR) tehniques [], as illustrated in Fig.. Depth is firstly used to find the orrespondene between a pixel of the image and points of the D world, based on the intrinsi and extrinsi parameters of the ameras. Then, the D point is projeted bak onto another viewpoint. When lossy ompression of the depth data is performed, the DIBR dereases its predition preision. More preisely, one pixel of the referene image is mapped to a segment of several pixels in the predited image (represented by in Fig. ). Due to this impreise mapping, the error in DIBR is diffiult to ontrol, whih leads to omplex ompression algorithms [7], [9], [0]. Depth maps have generally a high preision (6 levels) whih is not always neessary, as in the ase of simple predition for example. When a simple lossy oding of depth is performed, one preserves an unneessary bit

2 view r depth D point D sene disparity ' view Fig.. Illustration of the differene between depth-based predition and disparity ompensation. depth preision while it introdues losses in ruial regions. In that sense, depth-based representation does not enable a natural ontrol of geometry ompression error. In our GBR representation, we therefore propose another desription of geometry information, whih onstitutes a simpler version of depth and whih is eventually losslessly oded. In that way, this geometry signal orresponds to the minimum information that is needed at the deoder for view predition. During view predition, one pixel of the referene frame is shifted of D pixels in the predited viewpoint, with: D = f.d Z, () where f is the foal length, d the distane between the two ameras and Z the depth attahed to this pixel. In Fig., we plot the integer disparity D as a funtion of the depth Z whih takes 6 values between Z = 0 m and Z = 0 m, for different distanes between the referene image and the predited amera. We learly see that the disparity requires less preision than the original depth signals. We also remark that the number of levels inreases with the inter-amera distane d. In other words, the omplexity of the geometry signal that needs to be transmitted varies with the position of the predited view. Moreover, we remark that the number of disparity levels is larger for small depth values, whih also means that the geometry signal omplexity depends on the sene ontent. In our GBR representation, we propose a ompat representation of the disparity values, whih permit to reonstrut a predefined set of views. Contrary to traditional multiview oders, whih transmit one disparity value for bloks of pixels, our solution onsiders dense disparity maps. The GBR representation is detailed in the next setion. III. GRAPH-BASED GEOMETRY REPRESENTATION We reall here the main ideas of GBR onstrution proess and the view reonstrution at the deoder. Readers are referred to [] for details. Let us onsider a sene aptured by N ameras with the same resolution and foal length f. The n-th image is denoted by I n, with n N, where I n (r, ) is the pixel at row r and olumn. We only onsider translation between ameras, and we assume that the views are retified. In other words, the geometrial orrelation between the views I n is only horizontal. We assume that aurate depth images, Z n, are available at the enoding for every viewpoint, I n. As explained above, we ompute N dense disparity maps r D, disparity (pixels) 0 d = 0 m d = 0 m d = 0 m Z, depth (m) Fig.. Integer disparity as a funtion of depth for different distanes between the referene and predited amera. from these depth images. In what follows we assume that there are N predited images, whih are generated using the referene image along with struture and olor information introdued below. We ategorize the different types of pixels in terms of how they hange from one view to another. Beause of amera translation, a new part of the sene appears on the right or left of the image (appearing pixels) and another part disappears (disappearing pixels). During amera translation, foreground objets move faster than the bakground. As a result, some bakground pixels may appear behind objets (disoluded pixels). Conversely, some bakground pixels may beome hidden by a foreground objet (oluded pixels). If we onsider a pair of images (referene and target), a row of the target image an be reonstruted by opying pixels from the orresponding row of the referene image, exept when the abovementioned types of pixels our (in whih ase new pixels have to be inserted). Our graph approah diretly onveys this information by transmitting either i) a link to the loation in referene row where pixels should be opied from, or ii) the values of new pixels to be inserted. A graph with N levels desribes referene image and N predited ones. We show in Fig. a simple graph onstrution example, with levels ( referene and predited images). Its onstrution requires the depth maps Z n, n N. Sine the objet displaement is only horizontal, we onsider independent graph onstrution for eah image row. Suh a graph is made of two omponents, whih are desribed by two matries of size N W, where N is the number of levels (i.e., the number of images enoded by the graph) and W is the image width. These two matries are the olor values Γ r and the onnetions Λ r and represent olor and geometry information for all pixels of all images, where r is the row index (a pair of matries per row). Γ r and Λ r are generated based on the following priniples. Pixel intensity values are stored in the level (view) where they appear first. This means that a given level only ontains pixels that were not present in a lower level. The onnetions simply links these new pixels to the position of their neighbor in the previous level. We look now in more details at Fig.. First, the intensities of appearing pixels, (a), are stored in Γ r. No onnetivity information is needed, sine these pixels appear on the side Note that while we onstrut the graph row by row, ompression tehniques ould be developed that exploit redundanies aross rows.

3 foreground objet bakground level level level level level image image image image image row r Bakground disparity Foreground disparity a b d Conatenated rows Graph representation Fig.. Toy graph onstrution example: blue texture bakground has a disparity of at eah level and red retangle foreground a disparity of for eah level. Graph ontains all different types of pixels: a) appearing, b) disoluded, ) oluded and d) disappearing Reonstrution of level Fig.. Reonstrution of the level with the toy example of Fig.. Green arrows indiates the graph exploration order for view reonstrution. of the image. Disoluded pixels, (b), do not appear in the lower level, and their intensity is stored in the olor matrix Γ r. A set of onseutive disoluded pixels at level r starts right after a pixel that appears at level r. Thus, our graph links the first disoluded pixel at level r to the last opied pixel from level r (b). Oluded pixels, (), are pixels at level r that are not opied to level r. This situation is represented by links in the graph that go from level r to level r and bak to level r without inserting any pixel values. For example, in Fig. the links between the pixel at position in level, through level to the pixel at position 6 in level represent the olusion of the pixel at position in the representation at level. Finally, disappearing pixels, (d), are simply represented by a link (but no pixel intensity value) after the last pixel to be displayed. To get a different view on the graph representation, refer to Fig., where we show the image of level that is reonstruted based on the graph of Fig.. By reonstrution we mean reating an output row ontaining all pixel values at level based on the sparse graph representation. Reonstrution involves traversing the graph (left to right) and opying pixel values from either level or level to the output, following the links in the graph. In what follows, pixel numbering orresponds to their order in the reonstruted level shown in Fig.. The reonstrution starts with the appearing pixel at level. Then, it moves to the referene level and opies the orresponding pixels until enountering a link. In the ase of Fig., the first onnetion is after pixel and links it to pixels and in level, whih are disoluded pixels. After all disoluded pixels have been opied, the reonstrution goes bak to the referene level and opies (, 6 and 7) until the next non-zero onnetion (at pixel 7). The onnetion in 7 indiates an oluded region. Hene, the reonstrution algorithm jumps in the referene frame and restarts the filling proess (pixel 8 to 9) until the next nonzero onnetion (disappearing pixel). The reonstrution of the other levels is done reursively. Note that, in ontrast to depth-based representations, our GBR expliitly aptures the orrespondene between levels, making it easier to ontrol the desired level of quality in the representation. IV. EXPERIMENT In this setion, we show that the geometry sent with our GBR representation orresponds to the proper level of preision that is needed at the deoder side for inter-view predition, ontrary to depth-based sheme. More preisely, we show that GBR offers a higher ontrol of geometry loss impat on the reonstruted quality, ompared to depth representation. In the test presented below, we ode the geometry while the texture signal is not ompressed. We have implemented a prototype oding sheme for our GBR solution. As we an observe in the example of Fig., the matrix Λ r has a large number of zero values. We do not ode diretly Λ r and rather onsider a smaller matrix Φ of size M, where M is the number of non-zero elements in all the Λ r with r < H (H is the height of the image). The matrix Φ stores all the meaningful onnetions and it is organized as follows. The first olumn of Φ ontains this row indies r. The seond olumn ontains the olumn indies, the third olumn ontains the level indies, and finally, the fourth olumn ontains the onnetion values. Then, we simply onsider an arithmeti oding of every olumn, with, for some of them, a differential operation as preproessing, in order to derease the entropy. Alternatively, the depth maps are enoded using the lassial image oding tool JPEG000 []. They are used for view predition at deoder, using DIBR algorithm. As for GBR, no orretion is sent sine we only onsider goemetry ompression in these tests. In Fig., we show the predition error images using depth or GBR as geometry information. We only onsider views in this experiment. First, we build our GBR representation derived from the dense disparity values omputed with the depth images. These disparity map is designed for the predition

4 foreground in () is represented with several different depth values while one would be enough as it is the ase in (b). Moreover, some losses has been introdued in the depth-based oding sheme, for example around the foreground boundaries. In other words, the GBR representation permits to transmit the exat level of geometry required at the deoder side in ontrary to depth-based sheme, whih leads to more aurate view predition as shown by the PSNR gain observed for these datasets. The previous experiments show the validity of our solution and prove that GBR onstitutes a serious alternative to depth-based data representation shemes in multiview imaging. (a) Predition error with lossy depth (7. db) V. CONCLUSION In this paper, we have presented our new graph-based representation used to desribe geometry and texture information of multiview images. In addition to removing spatial redundanies in the data, it provides an intuitive graph struture that permits to effiiently represent the geometry signal. This leads to a better ontrol of inauraies due to geometry ompression, and their impat on the multiview reonstrution quality, ompared to depth-based approah. REFERENCES (b) Predition error with GBR information (.8 db) Fig.. Predition error images for sawtooth dataset using depth (a) and GBR (b). Coding rate of. kb for geometry rate. PSNR is alulated on the non oluded regions of the image. of view. The geometry size after the oding of the GBR data is. kb. We then enode the depth map with the same bitrate. For both deoded geometry information we perform the predition of view, we alulate the error image and we evaluate the PSNR on the non oluded regions. We see in Fig. that the GBR-based predition has error only on the disoluded regions, while depth oding introdues artifats also on predited regions. Next, we show that GBR adapts the omplexity of its geometry signal to the one of the predition proess. We still onsider lossless texture and only two views. For different distanes between the two views ( and times the intra-oular distane), we run the following test. We first build the GBR representation and we obtain a given geometry rate R in bits. Then, we ompress the depth map of view with the same rate. We then observe the deoded geometry information. They are presented for Venus and Sawtooth datasets in Fig. 6. We also show the predition PSNR values alulated on the non oluded regions. We see that when the distane inreases, the GBR provides higher preision. More preisely the foreground objets in (b) and (h) are desribed with more different values in respetively (e) and (k). At the same time, the depth signal keeps useless preision. More preisely, we see that the right [] Z. Zhang, Mirosoft kinet sensor and its effet, IEEE Multimedia, vol. 9, pp. 0, 0. [] C. Fehn, Depth-image-based rendering (dibr), ompression and transmission for a new approah on d-tv, Pro. SPIE, Stereosopi Image Proess. Render., vol. 9, pp. 9 0, 00. [] F. Shao, G. Jiang, M. Yu, and Y. Zhang, Objet-based depth imagebased rendering for a three-dimensional video system by olororretion optimization, Opt. Eng., vol. 0, pp , 0. [] P. Merkle, Y. Morvan, A. Smoli, D. Farin, K. M uller, P. de With, and T. Wiegand, The effet of depth ompression on multiview rendering quality, in D TV Conferene, Istanbul, Turkey, May 008. [] H. Yuan, Y. Chang, J. Huo, F. Yang, and Z. Lu, Model-based joint bit alloation between texture videos and depth maps for D video oding, IEEE Trans. on Cir. and Syst. for Video Tehnology, vol., pp. 8 97, 0. [6] B. Rajei, T. Maugey, and P. Frossard, Rate-distortion analysis of multiview oding in a DIBR framework, Annals of Teleommuniations, 0. [7] G. Cheung, W. Kim, A. Ortega, J. Ishida, and A. Kubota, Depth map oding using graph based transform and transform domain sparsiation, in IEEE Int. Workshop on Multimedia Sig. Pro., Hangzhou, China, Ot. 0. [8] I. Daribo, G. Cheung, and D. Florenio, Arithmeti edge oding for arbitrarily shaped sub-blok motion predition in depth video oding, in Pro. IEEE Int. Conf. on Image Proessing, Orlando, FL, USA, Sep. 0. [9] S. Kim and Y. Ho, Mesh-based depth oding for d video using hierarhial deomposition of depth maps, in Pro. IEEE Int. Conf. on Image Proessing, San Antonio, TX, USA, Sep [0] W. Kim, A. Ortega, P. Lai, T. D, and C. Gomila, Depth map oding with distortion estimation of rendered views, in Pro. of SPIE, the Int. So. for Optial Engineering, 00. [] T. Maugey, A. Ortega, and P. Frossard, Graph-based representation and oding of multiview geometry, in Pro. Int. Conf. on Aoust., Speeh and Sig. Pro., Vanouver, Canada, 0. [] JPEG-000, ISO/IEC FCD -: JPEG 000 final omitee draft version.0, 000. [Online]. Available: FCD-.htm

5 Original data GBR disparity map Corresponding ompressed depth (a) texture image (b) d =, R =.0kb,. db () d =, R =.0kb, 8. db (d) depth map (e) d =, R =.kb,.0 db (f) d =, R =.kb, 6.9 db (g) texture image (h) d =, R =.kb,.8 db (i) d =, R =.kb, 7. db (j) depth map (k) d =, R =.8kb, 7.9 db (l) d =, R =.8kb,.9 db Fig. 6. Illustration on Venus (a-f) and Sawtooth (g-l) multiview datasets of how the inter-view distane (d) impats on the geometry signals of both the GBR representation (b,e,h,k) and the depth-based approah (,f,i,l). This distane is expressed in view index (i.e., multiple of the inter-amera distane) and orresponds to or times the intra-oular distane (6. m). We expressed for every geometry map its rate R and the PSNR of the predition using it (on the non oluded regions).