MOTION-COMPENSATED COMPRESSION OF POINT CLOUD VIDEO

Transcription

1 MOTION-COMPENSATED COMPRESSION OF POINT CLOUD VIDEO R. L. de Queiroz Universidade de Brasilia Brasilia, Brazil P. A. Chou 8i Labs In. Bellevue, WA, USA ABSTRACT 3D immersive ommuniations are trendin as real-time point louds apture and display of point loud video beome feasible. This paper presents a novel motion-ompensated approah to enodin dynami voxelized point louds (VPC) at low bit rates. A simple oder breaks the VPC into bloks whih are intra-frame oded or replaed by a motion-ompensated version of a blok in the previous frame. The deision is optimized in a rate-distortion sense, enodin with distortion both eometry and the olor, at redued bit-rates. Inloop filterin is employed to minimize ompression artifats aused by distortion in the eometry information. Simulations reveal that this simple motion ompensated oder an effiiently extend the ompression rane of dynami voxelized point louds to rates below what intra-frame odin alone an aommodate, tradin rate for eometry auray. 1. INTRODUCTION In immersive 3D ommuniation systems [1]-[5], a dynami 3D sene apture an be implemented usin olor plus depth (RGBD) ameras, while 3D visualization an be implemented usin stereosopi monitors or near-eye displays to render the subjet within a virtual or aumented reality. The proessin for apture and display an be done in real time usin powerful raphis proessin units (GPUs) [6]. However, representin a omplex, dynami 3D sene enerates a lare amount of data. Compression is therefore an essential part of enablin these emerin immersive 3D systems for ommuniation. There are many hoies for representin 3D data, and the most appropriate hoie depends on the situation, for example, dense voxel arrays, polyonal meshes, or point louds. An alternative to point louds are sparse voxel arrays, or voxel louds, whih are arbitrary olletions of voxels with a volumetri aspet. Yet another possible representation of 3D data is simply a set of olor and depth maps, sometimes alled multiview video plus depth (MVD) [7]. This is a low-level representation lose to the RGBD sensors. Closely related to olor and depth maps are elevation maps [8] and multi-level surfae maps [9]. Here, the 3D representation of hoie is sparse voxel arrays, or voxelized point louds. Compared to arbitrary point louds, they have implementation advantaes and are hihly effiient for realtime proessin of aptured 3D data [6]. Eah representation employs its own ompression tehniques, suh as [10] [20] for polyonal meshes, and [7], [21] for multiview video plus depth. Previous approahes to ompressin point louds inlude [22] [28]. The most effiient of these are based on voxelization. We believe [25] and [26], until reently, represented the state-of-the-art in (voxelized) point loud olor ompression. We Fi. 1. Different viewpoints of a 3D point loud frame. have reently developed a oder that is able to math or outperform existin methods at a redued ost [29]. Suh a still-frame oder is based on a reion-adaptive hierarhial transform (RAHT) speially developed for point louds and is used as a fundamental buildin blok in the present framework for our dynami point loud oder, whih an be onsidered a 3D video oder [30]. 2. VOXELIZED POINT CLOUDS We represent 3D data by voxelized point louds. A point loud is a set of points {ν}, eah point ν havin a spatial position (x, y, z) and a vetor of attributes suh as olors, normals, or urvature. In this paper we assume the attributes are olors, ()e.. Y, U, V ). A point loud may be voxelized by quantizin the point positions to a reular lattie. A quantization ell, or voxel, is said to be oupied if it ontains a point in the point loud and is unoupied otherwise. An oupied voxel derives its olor from the olor(s) of the point(s) within the voxel, possibly by averain, but this is outside the sope of this paper. We assume simply that eah oupied voxel has a olor. Without loss of enerality, we may assume that the voxels are addressed by positions in the inteer lattie Z 3 W, where Z W = {0,..., W 1}, W = 2 D is its width, and D is an inteer. We take Y, U, and V to be 8-bit unsined inteers. Thus, our voxelized point loud is a finite set or arbitrarily indexed list of oupied voxels {ν i} in whih eah voxel ν i = [x i, y i, z i, Y i, U i, V i] (1) omprises a unique inteer spatial loation (x i, y i, z i) and an inteer olor vetor (Y i, U i, V i). We are mostly interested in live video and dynami point louds. Thus at every disrete time t we have the frame F(t) = {ν it}, whih is represented as a list of voxels ν it = [x it, y it, z it, Y it, U it, V it]. (2) Note that different frames may be represented by different lists of voxels, so there is no real relation between ν i,t and ν i,t+1, sine the indexin of the voxels in the lists is arbitrary. Moreover, different frames may have different numbers of oupied voxels. The temporal orrespondene amon voxels of adjaent frames is not treated here, and left for the fuller version of this paper [30].

2 A lare amount of work exist in the subjet bein found for example in [31],[33]-[41] It is, in eneral, a very omplex proess that is not friendly to real-time ommuniations. 3. DISTORTION METRICS AND RESIDUALS For the purposes of visual reonstrution, the unoupied and interior voxels do not need to be enoded, but only the external shell of the person or objet. The sparsity of these oupied voxels allows effiient still frame or intra-frame ompression of the eometry usin ottrees [6],[28]. Inter-frame ompression of the eometry an also be performed usin ottrees and exlusive-or between sets of oupied voxels in suessive frames [26],[28]. Here, we attempt to predit the eometry as well as the olor, and to ode the predition residuals in a rate-distortion optimized way. The eometry distortion (ombined or not with the olor distortion) has not been well defined yet. We an devise a orrespondeneand a projetion-based distortion omputation. In a orrespondene-based distortion metri, we first establish a orrespondene between the oriinal frame F(t) and the reonstruted frame ˆF(t) usin for example proximity, i.e., a voxel in ˆF(t) is assoiated to its spatially losest voxel in F(t). From all pairin assoiations we an ompute the mean-squared error (MSE) for both eometry and olors, and, from it, the orrespondin peak sinal-to-noise ratio (PSNR). In partiular, the MSE we use only aounts for the Y omponent and linearly blends both distortion measures (δ and δ ). δ Y +G = δ + βδ. (3) From δ Y +G we ompute PSNR-Y+G. In a projetion-based distortion measure, a projetion view of the point loud is rendered for a viewer. We assume an orthoonal projetion of the point loud over the six sides of a ube at the limits of the voxel spae. The observer is assumed far away from the sene so that the rays from it to the voxels are parallel and the bakround is assumed at a mid level of ray. The distortion metri is the MSE (or PSNR) in between the two orrespondin omposite imaes, eah with the 6 orthoonal projetions of the oriinal or reonstruted Y frames. 4. THE MOTION-COMPENSATED CODER Our objetive is to build a oder for dynami point louds that an be implemented in real time with existin tehnoloy, whih would outperform the use of RAHT and ottrees to ompress olor and eometry, respetively. Unlike previous approahes, we introdue the use of motion estimation and motion ompensation into the ompression of dynami point louds, in order to ahieve hiher ompression ratios at the expense of lossy odin of the eometry. As far as we know, the present work is the first to explore suh a framework Cube motion ompensation The oder, whose diaram is depited in Fi. 2, is very similar to any traditional video oder in its essene, but is quite different in the details. The frame is broken into bloks of N N N voxels. So, the oupied blok at inteer position (b x, b y, b z) is omposed of oupied voxels ν it within the blok boundaries, i.e., oupied voxels ν it = [x it, y it, z it, Y it, U it, V it] suh that b xn x it < b xn + N, b yn y it < b yn + N, b zn z it < b zn + N. (4) Fi. 2. Enoder diaram. Fi. 3. Motion ompensation of an oupied blok in the present frame with voxel data within a translated blok in the referene frame. An oupied blok may have between 1 and N 3 oupied voxels. The motion ompensation proess is illustrated in Fi. 3. Eah oupied blok is assoiated with a motion vetor (MV), whose omponents (M x, M y, M z) indiate a blok in a referene frame that will be used to predit the urrent blok. Let Ω be the set of oupied voxels in a blok at position (b x, b y, b z) in frame F(t). Then, Ω an be predited from the set of voxels [x i,t 1, y i,t 1, z i,t 1, Y i,t 1, U i,t 1, V i,t 1] oriinally in frame F(t 1) suh that b xn M x x i,t 1 < b xn + N M x, b yn M y y i,t 1 < b yn + N M y, b zn M z z i,t 1 < b zn + N M z. This set is motion ompensated by addin the motion vetors to its oordinates (x i x i + M x, y i y i + M y, and z i z i + M z) to obtain the set Ω p of voxels [x i,t 1 + M x, y i,t 1 + M y, z i,t 1 + M z, Y i,t 1, U i,t 1, V i,t 1]. The set Ω p is used as a preditor of Ω. In order to ompute a loal distortion δ between Ω and Ω p, with N Ω and N Ωp voxels, respetively, we set δ = 0 and ompute all N ΩN Ωp Eulidean distanes aross the sets. We assoiate the voxels with the shortest distane, remove them, and update δ δ + δ + βδ, where δ and δ are the eometry and olor distanes. We repeat the proess until all voxels in Ω are one. If N Ωp < N Ω, we may dupliate voxels in Ω p beforehand. If we opt for a projetionbased distortion, the MSE in between the projetions of Ω and Ω p is (5)

3 omputed instead. In this artile, we do not examine how best to perform motion estimation to establish what the MVs are. Instead we re-use the orrespondenes that are alulated in the 3D surfae reonstrution proesses immediately prior to ompression [41]. In those proesses, eah voxel may have a orrespondene to a voxel in the previous frame, but we need to use one MV per oupied blok. From the orrespondenes, we first produe a voxel-oriented field of MVs. In order to find one MV for the whole set, we take the existin MV in Ω that is the losest to the averae of all MVs in Ω. That median MV is then assined to the blok ontainin Ω Codin mode deision Geometry predition and residuals are a distint problem. We, so far, have been unable to enode the eometry predition-error at a rate sinifiantly lower than bpv, whih is ahieved by enodin the eometry usin ottrees without any inter-frame predition. Beause of that we do not enode eometry residuals. Our oder operates in two modes: either a blok is purely motion ompensated or it is entirely enoded in intra mode. We desinate frames as types I (intra-oded) and P (predited). For an I-frame, all bloks are enoded in intra mode, for example usin ottree enodin for the eometry and RAHT enodin for the olor omponents. For a P-frame, there should be a referene frame stored in the frame store, typially the previous frame. In a P- frame, we make a mode deision for eah oupied blok, whether it should be inter-oded (motion-ompensated) or intra-oded. In effet, we test whether motion ompensation alone produes a ood enouh approximation of the blok. If so, we replae Ω by Ω p. If not, we enode Ω independently usin ottree and RAHT enodin. The deision is optimized in a rate-distortion (RD) sense. The hoie is about representin the blok by Ω (intra) or by Ω p (inter). Eah hoie implies rates and distortions for both eometry and olor omponents: (R intra R inter, D inter, and D inter R intra = R intra, R intra, D intra, D intra ) and (R inter ). We then ompute + R intra R inter = R inter D intra = D intra D inter = D inter, 2.5 Ω + R intra (6) + R inter = R MV (7) + βd intra = βd intra (8) + βd inter = δ, (9) where R MV is the averae rate to enode one MV. We build Laranian osts for eah mode and hoose the mode with the smallest ost, i.e., we deide on intra mode if and only if D intra + λr intra < D inter + λr inter (10) and deide on inter mode otherwise, for any fixed λ > 0. The points (R inter, D inter) and (R inter, D inter) are very distint points in the distortion-rate plane, with (typially) R inter < R intra and D inter > D intra. Let λ = Dinter Dintra R intra R inter > 0 (11) be the manitude of the slope of the line onnetin the two points. Then, the intra mode riterion (10) redues to λ < λ. That is, a blok is enoded as intra if and only if its value of λ is reater than the lobally advertised value of λ, whih is fixed aross the sequene. One an see that the mode deision for a iven blok is not exessively sensitive to λ sine the hoie is between only two RD points and many values of λ may lead to the same mode deision for a iven blok Rate or distortion ontrol The rates and distortions of the olor and motion vetors are ontrolled by a quantizer step Q. Like λ, Q is also a means to trade off rate and distortion. Like similar video oders, the overall oder essentially maps Q and λ to an overall rate R and distortion D. We would like the oder to operate on the lower onvex hull (LCH) of all the RD points produed by spannin all Q and λ ombinations. Thus, we want to find the λ and Q points that are mapped into the LCH. We simplify the proess orrelatin λ and Q as λ = f λ (Q). Details an be found in [30] In-loop proessin for eometry distortions Unlike previous oders for dynami point louds, in this work, we apply lossy odin of the eometry. Even thouh our distane metri applied to two sets of point louds may be useful as an objetive measurement of the odin quality, small distortions to the eometry an ause blokin artifats that are quite annoyin. We deided to smooth the surfaes and to fill the aps (rips) usin adaptations to 3D voxels of traditional operators. Our proessin is essentially an in-loop deblokin filter adapted to 3D voxels. We first filter the eometry elements to smooth the surfae disontinuities aused by mismath in the motion ompensation proess. Without loss of enerality, for example, usin the first dimension (x), the filter is: ˆx i = j,d ij <η xjρd ij j,d ij <η ρd ij (12) where d ij = ν i ν j is the distane between voxels i and j, and η ontrols the neihborhood size and the intensity of filterin. Suh an operation may ause further holes in the eometry surfaes. Beause of that, assumin the disontinuities will be more prominent at the ube boundaries, we only replae voxels that are near the boundaries of a motion-ompensated ube. Furthermore, to avoid reatin more holes, we do not allow the voxel position to move away from the border. In effet, if x i is at the border, x i is not haned but y i and z i are replaed by ŷ i and ẑ i, respetively. After filterin, we try to lose aps usin morpholoial operations on the voxel eometry. We define simple dilation as repliatin eah existin oupied voxel to its 26 volumetri neihbors, if suh a neihbor is not oupied yet. We also define simple erosion by erasin a iven voxel if any of its 26 neihbors in 3D is unoupied. We repeat the dilation γ times and do the same number of erosion operations. The proess is parallel to a morpholoial losin operation. Holes up to 2γ voxels wide may be pathed by the proess. The operation is only applied to inter-oded ubes. Proessed ubes not only are made available to the deoder and to the display, but an also be plaed in the frame store so that the oder an use the frames proessed in-loop to perform motion ompensation for future frames. 5. SIMULATION RESULTS Simulations were arried out to demonstrate the apabilities of the proposed system, to whih we refer as a motion-ompensated intraframe oder (MCIC). We have two datasets, both with W = 512, 200 frames, and aptured at a rate of 30 frames per seond. One sequene ( man ) represents a full body in 3D, while the other sequene ( Riardo ) is intended for video onferenin and thus just a frontal upper body is represented. We used a GOP lenth of 8, i.e. interspersin 7 P-frames between every I-frame. Sine P-frames are deraded versions of I-frames (lower rate and hiher distortion) and

4 44 Sequene Man Level 9: f1-200 PSNR-Y+G (YMSE GeomMSE) MCIC All intra Rate in bpv (Y+U+V+Geometry) 46 Sequene Riardo Level 9: f1-200 PSNR-Y+G (YMSE GeomMSE) MCIC All intra Rate in bpv (Y+U+V+Geometry) Fi. 4. RD plots for sequene man and upper body usin PSNR- Y+G. RD points are averaes over the entire sequene. Fi. 5. Front projetion renderin omparison of frame 58 in sequene man at 3.7 bpv. Oriinal, MCIC, MCIC desined under a projetion-based distortion metri, RAHT. assumin the ompression ratio should be similar for every intraoded part in every frame, the rate peaks at every I-frame. We used Q mv = 1 and varied Q to obtain our RD urves. Values of Q in the rane of 5 throuh 45 were used for the MCIC, while the purely intra oder used quantizer values ranin from 10 to 50. As for the in-loop filterin, after many tests, we have hosen to use γ = 2 and η = 2. Nevertheless the best hoie of parameters and the filterin approah is far from bein settled. For the orrespondene-based distortion metri (PSNR-Y+G), a simple approximation to f λ (Q) yields very ood results for both sequenes we tested. We derived the funtion from one sinle frame and yet it performs adequately for all other frames under this metri. Usin λ = f λ (Q) = Q 2 /60, we obtained the RD plots for sequenes man and Riardo shown in Fi 4. The RD points are averaes over all the 200 frames of the sequenes. From the fiure, one an easily infer the superior performane of the MCIC over purely intra oder under this metri. We do not have room to present and disuss the results usin a projetion-based distortion metri (PSNR-P), whih were left to the full version of the present paper [30]. We also inlude a similar omparison for a frame of sequene Riardo, whih was ompressed usin MCIC (orrespondenebased distortion for mode seletion) and intra odin, at a rate around 2.6 bpv, whih is shown in Fi. 6. Fi. 6. Front projetion renderin omparin MCIC (riht) and intra-oder (RAHT, left). Frame 60 at 2.6 bpv. 6. CONCLUSIONS We have developed a novel motion-ompensation sheme for use with dynami point louds. The enoder works on dividin the loud into bloks of oupied voxels and deidin for eah one if the blok should be intra-oded or simply motion-ompensated from the past frame. The replaement of intra-oded data for a slihtly distorted (or not) set of voxels saves many bits, but introdues errors not only to the voxel olors, but also also to their positions (eometry). In effet, a P-frame is a deraded I-frame whose extra distortion is found to be worth it in an RD sense. With that extra deradation we are able to extend the bit-rate rane below where the intra oder an effetively operate and to exeed the performane of the intra oder at any rate under a iven objetive distortion measure. From that perspetive, the oder results are satisfatory. However, muh still needs to be done.

5 7. REFERENCES [1] J. Lanier, Virtually There, Sientifi Amerian, pp , Apr [2] Y. Altunbasak, et al., Realizin the vision of immersive ommuniation, IEEE Sinal Proessin Maazine, Jan [3] P. A. Chou, Advanes in immersive ommuniation: (1) telephone, (2) television, (3) teleportation, ACM Trans. Multimedia Comput. Commun. Appl., vol. 9, no. 1s, [4] J. G. Apostolopoulos, et. al., The road to immersive ommuniation, Pro. IEEE, vol. 100, no. 4, pp , [5] C. Zhan, et al., Viewport: a fully distributed immersive teleonferenin system with infrared dot pattern, IEEE Multimedia, vol. 20, no. 1, pp , [6] C. Loop, C. Zhan, and Z. Zhan, Real-time hih-resolution sparse voxelization with appliation to imae-based modelin, in Hih-Performane Graphis Conf. pp , [7] P. Merkle, et al., Multi-view video plus depth representation and odin, in IEEE Int. Conf. Imae Proessin, [8] M. Hebert, et al., Terrain mappin for a rovin planetary explorer, in Pro. IEEE Int. Conf. Robotis & Automation, [9] R. Triebel, P. Pfaff, and W. Burard, Multi-level surfae maps for outdoor terrain mappin and loop losin, in Pro. IEEE Int. Conf. Intellient Robots and Systems, [10] O. Devillers and P. Gandoin, Geometri ompression for interative transmission, in IEEE Visualization, pp , [11] X. Gu, S. J. Gortler., and H. Hoppe, Geometry imaes, in Annual Conf. Computer Graphis and Interative Tehniques, pp , [12] H. M. Brieno, et al., Geometry Videos: A New Representation for 3D Animations, in ACM Symp. Computer Animation, pp , [13] S. Gupta, K. Senupta, and A. A. Kassim, Reistration and partitionin-based ompression of 3D dynami data, IEEE Trans. Ciruits Syst. Video Tehn., vol. 13, no. 11, pp , Nov [14] H. Habe, Y. Katsura, and T. Matsuyama, Skin-off: Representation and ompression sheme for 3D video, in Piture Codin Symp., 2004, pp [15] J. Pen, Chan-Su Kim, and C. C. Jay Kuo, Tehnoloies for 3D mesh ompression: A survey, J. Vis. Comun. and Imae Representation, vol. 16, no. 6, pp , De [16] S.-R. Han, T. Yamasaki, and K. Aizawa, Time-varyin mesh ompression usin an extended blok mathin alorithm, IEEE Trans. Ciruits Syst. Video Tehn., vol. 17, no. 11, pp , Nov [17] L. Váša and V. Skala, Geometry driven loal neihborhood based preditors for dynami mesh ompression, Comput. Graph. Forum, vol. 29, no. 6, pp , [18] R. Mekuria, et al., A 3D Tele-Immersion System Based on Live Captured Mesh Geometry, ACM Multimedia Systems Conferene, Oslo, Feb [19] H. Q. Nuyen, P. A. Chou, and Y. Chen, Compression of human body sequenes usin raph wavelet filter banks, in IEEE Int. Conf. Aoustis, Speeh, and Sinal Proessin, pp , May [20] J. Hou, et al., Compressin 3D human motions via keyframebased eometry videos, IEEE Trans. Ciruits Syst. Video Tehn., vol. 25, no. 1, pp , [21] W. Sun, et al., Rate-onstrained 3D surfae estimation from noise-orrupted multiview depth videos, in IEEE Trans. Imae Proessin, Jul [22] T. Ohotta and D. Saupe, Compression of point-based 3D models by shape-adaptive wavelet odin of multi-heiht fields, in Euroraphis Conf. on Point-Based Graphis, 2004, pp [23] R. Shnabel and R. Klein, Otree-based point-loud ompression, in Symposium on Point-Based Graphis, Jul [24] Y. Huan, et al., A eneri sheme for proressive point loud odin, IEEE Trans. Vis. Comput. Graph., vol. 14, no. 2, pp , [25] C. Zhan, D. Florênio, and C. Loop, Point loud attribute ompression with raph transform, in IEEE Int. Conf. Imae Proessin, Sep [26] D. Thanou, P. A. Chou, and P. Frossard, Graph-based motion estimation and ompensation for dynami 3D point loud ompression, in IEEE Int. Conf. Imae Proessin, Sep [27] D. Thanou, P. A. Chou, and P. Frossard, Graph-based ompression of dynami 3D point loud sequenes, in IEEE Trans. Imae Proessin, to appear. [28] J. Kammerl, et al., Real-time ompression of point loud streams, in IEEE Int. Conf. Robotis and Autom., May [29] R. de Queiroz and P. Chou, Compression of 3D point louds usin a reion-adaptive hierarhial transform, IEEE Trans. on Imae Proessin, Vol. 25, pp , Au [30] R. de Queiroz and P. Chou, Motion-ompensated ompression of dynami voxelized point louds, preprint. [31] Y. Wan, et al, Handlin olusion and lare displaement throuh improved RGB-D sene flow estimation, IEEE Trans. Ciruits Syst. Video Tehn., to appear. [32] A. L. Yuille and N. M. Grzywaz, A mathematial analysis of the motion oherene theory, Int l J. of Computer Vision, vol. 3, no. 2, pp , June [33] S. Hadfield and R. Bowden, Sene partiles: Unreularized partile based sene flow estimation, IEEE Trans. Pattern Analysis and Mahine Intelliene, vol. 36, no. 3, pp , [34] J.-M. Gottfried, J. Fehr, and C. S. Garbe, Computin rane flow from multi-modal Kinet data, In Advanes in Visual Computin, paes Spriner, [35] E. Herbst, X. Ren, and D. Fox, RGB-D flow: Dense 3-D motion estimation usin olor and depth, In IEEE Int l Conf. on Robotis and Automation (ICRA), [36] J. Quiroa, F. Devernay, and J. L. Crowley, Loal/lobal sene flow estimation, In IEEE Int l Conf. on Imae Proessin (ICIP), [37] X. Zhan, et al., Dense sene flow based on depth and multihannel bilateral filter, in Spriner Computer Vision (ACCV), pp , [38] M. Hornaek, A. Fitzibbon, and R. Carsten, Sphereflow: 6 DoF sene flow from RGB-D pairs, in IEEE Int l Conf. on Computer Vision and Pattern Reonition (CVPR), [39] J. Quiroa, et al., Dense semi-riid sene flow estimation from RGBD imaes, in Spriner Computer Vision (ECCV), pp , [40] Y. Niu, A. Dik, and M. Brooks, Compass rose: A rotational robust sinature for optial flow omputation, IEEE Trans. Ciruits and Systems for Video Tehnoloy, vol. 24, no. 1, pp , Jan [41] M. Dou, et al., 3D Sannin Deformable Objets with a Sinle RGBD Sensor, in IEEE Int l Conf. on Computer Vision and Pattern Reonition (CVPR), 2015.