IN TRADITIONAL 2-D videos, users can only see a signal

Transcription

1 IEEE TANSACTIONS ON CICUITS AND SYSTEMS FO VIDEO TECNOOGY, VO. 1, NO. 4, API Moel-Base Joint Bit Allocation Between Texture Vieos an Depth Maps for 3-D Vieo Coing ui Yuan, Stuent Member, IEEE, Yilin Chang, Junyan uo, Fuzheng Yang, Member, IEEE, an Zhaoyang u Abstract In 3-D vieo coing, texture vieos an epth maps nee to be jointly coe. The istortion of texture vieos an epth maps can be propagate to the synthesize virtual views. Besies coing efficiency of texture vieos an epth maps, joint bit allocation between texture vieos an epth maps is also an important research issue in 3-D vieo coing. First, we present comprehensive analyses on the impacts of the compression istortion of texture vieos an epth maps on the quality of the virtual views, an then erive a concise istortion moel for the synthesize virtual views. Base on this moel, the joint bit allocation problem is formulate as a constraine optimization problem, an is solve by using the agrangian multiplier metho. Experimental results emonstrate the high accuracy of the erive istortion moel. Meanwhile, the rateistortion -D performance of the propose algorithm is close to those of search-base algorithms which can give the best -D performance, while the complexity of the propose algorithm is lower than that of search-base algorithms. Moreover, compare with the bit allocation metho using fixe texture an epth bits ratio 5:1, a maximum 1. B gain can be achieve by the propose algorithm. Inex Terms 3-D vieo coing, istortion moel, joint bit allocation, rate istortion optimization. I. Introuction IN TADITIONA -D vieos, users can only see a signal view, an cannot enjoy 3-D scenes. The emerging 3-D 3-DV, as the next generation visual technology, can provie arbitrary views of 3-D scenes for observers. Users can enjoy 3-D scenes without wearing special glasses, an can choose any view of a 3-D scene freely. 3-DV is the visual content of the well-known 3-D television 3-DTV [1] an free Manuscript receive February 8, 010; revise August 16, 010; accepte October 11, 010. Date of publication March 10, 011; ate of current version April 1, 011. This work was supporte in part by the National Natural Science Founation of China, uner Grants , , an , in part by the Funamental esearch Funs for the Central Universities, uner Grant K an , in part by the International Science an Technology Cooperation Program of China China- Finlan Joint Project, uner Grant 010DFB10570, in part by the 111 Project, uner Grant B08038, an in part by the National Basic esearch Program of China, 973 Program, uner Grant 009CB This paper was recommene by Associate Eitor. Onural.. Yuan is with the School of Information Science an Engineering, Shanong University, Jinan 50100, China, the State Key aboratory of Integrate Services Networks, Xiian University, Xi an , China, an the isense State Key aboratory of Digital Multi-Meia Technology Company, t., Qingao 66061, China yuanhui035@gmail.com. Y. Chang, J. uo, F. Yang, an Z. u are with the State Key aboratory of Integrate Services Networks, Xiian University, Xi an , China ylchang@xiian.eu.cn; jyhuo@mail.xiian.eu.cn; fzhyang@mail.xiian.eu.cn; zhylu@xiian.eu.cn. Digital Object Ientifier /TCSVT /$6.00 c 011 IEEE viewpoint television []. It can be use in many application fiels, such as visual ity, computer games, sports matches, surveillance, an so on. eal 3-D scenes must be efficiently represente by using a small amount of ata in orer to provie arbitrary views without acquisition of all views. There are a lot of scene representation technologies as summarize in [3]. Among all those technologies, texture-plus-epth representation has been extensively stuie because of its low processing cost. Virtual views can be synthesize from the acquire texture vieos an their corresponing epth maps, which is known as epth image-base renering [4] technology. Although the generation of epth maps is complex, the texture-plusepth representation format is compatible with the current legacy evices an existing elivery facilities for -D vieo. Furthermore, the texture-plus-epth representation format can also be compresse by the popular vieo coing stanars, such as.64/moel-view-controller MVC [5]. The quality of synthesize virtual views can be affecte by the compression of texture vieos an epth maps. Besies coing efficiency of texture vieos an epth maps, ifferent bits allocation between texture an epth can also affect the quality of virtual views when total bits of texture vieos an epth maps are restricte. In orer to allocate bits between texture vieos an epth maps, fixe texture an epth bits ratio 5:1 bit allocation was use in [4], but it cannot guarantee that it is optimal for 3-D renering. Morvan [6] propose a full search algorithm to fin the optimal quantization parameter QP pair for texture vieos an epth maps. In Morvan s algorithms, it is assume that a view exists at the synthesis position, an the istortion of the synthesize virtual view is evaluate by mean square error between the synthesize virtual view an its corresponing view. In [7], iu propose a istortion moel to estimate the istortion of virtual views without comparing the virtual view with its corresponing view. iu eeme that the istortion of synthesize virtual views is compose of three aitive istortions: vieo coing-inuce istortion, epth quantization-inuce istortion, an the inherent geometry istortion. Base on the estimate istortion, the bit allocation between texture vieos an epth maps is also performe by the search. Although the algorithms of Morvan [6] an iu [7] can fin the optimal or near optimal bit allocation between texture vieos an epth maps, the application of those algorithms is restricte by the high complexity.

2 486 IEEE TANSACTIONS ON CICUITS AND SYSTEMS FO VIDEO TECNOOGY, VO. 1, NO. 4, API 011 name as isparity. There exists a fixe relationship between isparity an epth value [11] Z = fl/ 1 Fig. 1. Block iagram of a 3-DV system. In orer to solve the bit allocation problem optimally with low complexity, the istortion of synthesize virtual views affecte by the compression of texture vieos an epth maps is analyze in etail, an a concise istortion moel of synthesize virtual views is erive. Then, the bit allocation problem is moele as a constraine optimization problem an is solve by using agrangian multiplier metho. Compare with those search-base bit allocation algorithm, the avancement of the propose algorithm is that the bit allocation problem is represente as a close-form formulation, an the optimal solution coul be calculate mathematically. Besies, it is no nee to compare the synthesize virtual view with its corresponing view to compute the istortion of the synthesize virtual view, or to estimate the istortion of synthesize virtual views. Only moel parameters shoul be preicte by pre-encoing texture vieos an epth maps. Therefore, the optimal bit allocation problem coul be solve by the propose algorithm with low complexity. The remainer of this paper is organize as follows. Backgroun of 3-DV system is outline in Section II. The effects of texture an epth compression on the istortion of virtual views are iscusse in Section III, an then the istortion moel of synthesize virtual views is evelope. Base on the istortion moel, the bit allocation problem is solve as a constraint optimization problem in Section IV. Experimental results an conclusions are presente in Section V an VI, respectively. II. Backgroun 3-DV is a complex system which is compose of ifferent moules, such as texture vieos acquisition an rectification incluing camera calibration an color correction among inter views, epth estimation or acquisition, joint texture an epth coing, transmission/storage, view synthesis an isplay, an others, as shown in Fig. 1. Texture vieos are acquire by camera array which contains several cameras with the same properties an a certain interval name as baseline in horizontal or vertical irection. Geometry iscrepancy among those acquire texture vieos can be correcte via camera calibration [8]; meanwhile, camera parameters can be calculate. Color iscrepancy of all the acquire texture vieos can be correcte via color compensation [9]. Depth maps can be acquire by epth cameras [10] or by estimation. Estimation-base epth generation is extensively use for its low cost. The principle of epth estimation is shown below. A point in 3-D scene can be projecte to ifferent positions in ifferent image planes. The position ifference is where Z is epth, f represents the focal length of cameras, is isparity, an l is the baseline between two cameras. Thereby, epth can be generate from ajacent texture vieos by using isparity estimation. Generally, a epth value is in floating-point type. In orer to compress epth using the existing coing framework such as.64, epth values must be quantize to epth map wherein pixel values are from 0 to 55. Consiering epth characteristics, non-uniform quantization is use to map epth values to the range of 0 55 [4], [1] = 55 1 Z 1 Z far Z near Z far where Z near an Z far are the values of the nearest an farthest epth of the scene, respectively, Z is the epth value of a point, is a floor operator, an is the mappe value by non-uniform quantization. Texture vieos an epth maps can then be compresse by the existing vieo coing stanars such as.64/mvc. The terminal of a 3-DV system first ecoes the compresse ata, an then synthesizes virtual views accoring to the requirements of the users. 3-D-warping [4], [13] is usually use to synthesize virtual views. The essence of 3-D warping is coorinate conversion. Thus, a pixel in the left view coul be projecte to another coorinate in the right view. The principle of 3-D-warping is briefly presente below. A pixel m in an image plane can be projecte to a point M in the 3-D worl coorinate X, Y, Z T = A 1 x, y, 1 T Z + t 3 where X, Y, Z T is the coorinate of M, x, y, 1 T is the homogeneous coorinate of pixel m, an can be represente as m, A an are the intrinsic an extrinsic parameter matrices of the camera, respectively, t=t x, t y, t z T is the coorinate of optical center of the camera, an is usually referre as translation vector since it can istinguish vertical or horizontal position ifferences between ifferent cameras in the same plane. Point M can also be projecte to the pixel m in a neighboring image plane Fig. s x,y, 1 T = A 1 [ X, Y, Z T t ] 4 where x, y, 1 T is the homogeneous coorinate of m an can be represente as m, s is a scale factor, an A,, an t are the corresponing intrinsic parameter matrix, extrinsic parameter matrix, an translation vector of the neighboring camera, respectively. For a well-calibrate camera system, A is the same as A, is the same as, an there exists only horizontal or vertical ifference between t an t. The corresponence between m

3 YUAN et al.: MODE-BASED JOINT BIT AOCATION BETWEEN TEXTUE VIDEOS AND DEPT MAPS FO 3-D VIDEO CODING 487 Fig.. Pixel m can be projecte to M in the 3-D worl coorinate system, an M can also be projecte to m. an m is shown in 5 which can be erive from 3 an 4 s m = Z m +A 1 [ t t ]. 5 Therefore, 6 can be further obtaine from 5 s x,y, 1 T = Z x, y, 1 T +A 1 T 6 t x t x,t y t y,t z t z. In practical applications, a virtual view an neighboring views are usually horizontally arrange. Thereby, for a wellcalibrate camera system, there only exists ifference in horizontal coorinate between t an t; 6 can be rewritten as s x,y, 1 T = Z x, y, 1 T +A 1 t x t x, 0, 0 T. 7 From 7, s can be euce to be equal to Z, y can be further euce to be inepenent with Z, an only x changes with Z s = Z, y = y, x = x + 8 Z where can be euce to be equal to f l base on the constitution of matrix A an in a well-calibrate camera system, such as the one shown in [14]. Thus, / Z equals to, an then it can be conclue that virtual views can be synthesize by isparity compensation when all the cameras are well calibrate. Practically, the left an right views of the virtual view are both use to eliminate the occlusion effect I V x, y = ω I x,y + ω I x,y 9 where I v x, y is the pixel value of the virtual view, I x, y an I x, y are the pixel values of the corresponing pixels in left an right views, ω an ω are two coefficients which satisfy ω +ω, an ω when pixel x, y are only visible in the left view, ω =1 when pixel x, y are only visible in the right view. From the above introuction an analyses, we can conclue that the coorinate conversion epens on the accuracy of epth maps an camera parameters. Besies, the istortion of neighboring texture vieos coul be propagate to the virtual views by 9. Therefore, the istortion of the virtual views coul be affecte by the accuracy of epth maps, camera parameters, the istortion of neighboring texture vieos, an the color iscrepancies between neighboring views. The 3-D isplay [15] moule can give users 3-D sensation, an can interact with users. In this paper, we only investigates Fig. 3. Virtual view s istortion calculation. joint bit allocation between texture vieos an epth maps, therefore, texture vieos acquisition, epth generation, virtual view synthesis algorithm, an 3-D isplay moules are assume to be fixe, an the performance of 3-DV system can only be affecte by the joint texture/epth coing moule. Similar to the existing bit allocation algorithms in [6] an [7], the performance of joint texture/epth coing is evaluate as the quality of the synthesize virtual views uner a certain bit constraint. The istortion of synthesize virtual views inuce by the compression of texture vieos an epth maps is analyze an verifie in the following section, an an optimal bit allocation algorithm between texture vieos an epth maps is propose base on the following analyses. III. Distortion Moel of Synthesize Virtual Views In 3-DV applications, a virtual view is synthesize from the compresse texture vieos an epth maps. The istortion of texture vieos an epth maps can then be propagate to the virtual views. Accoring to the assumption that texture acquisition, epth generation, view synthesis, an 3-D isplay moules in Fig. 1 are fixe, the istortion of virtual views can only be affecte by three factors, i.e., texture compression, epth compression, an the inherent epth inaccuracy inuce by epth estimation. During the analysis, we assume that there exists a view in the synthesis position, as shown in Fig. 3 [16], camera systems are well calibrate, an there is no color iscrepancy in all the acquire views, which means that the virtual view which is synthesize from uncompresse texture vieos an epth maps by 3-D warping is the same with the view in the synthesis position central view in Fig. 3 when epth maps are accurate enough. The istortion of a synthesize virtual view is first ecompose to three aitive items in Section III-A of this section, an each item is further analyze in Section III-B. The istortion moel of virtual view is finally constructe in Section III-C consiering the inherent epth inaccuracy. A. Decomposition of Virtual View s Distortion The istortion of a virtual view can be written D v I v x, y Î v x, y 10 x=0 y=0

4 488 IEEE TANSACTIONS ON CICUITS AND SYSTEMS FO VIDEO TECNOOGY, VO. 1, NO. 4, API 011 where D v represents the istortion of the virtual view inuce by the compression of texture vieos an epth maps, W an are the with an the height of the image, I v x, y an Î v x, y represent the pixel values at position x, y in the central an the virtual view, respectively. et Avg enote average operation of all the ata samples, an taking 9 into account, 10 can be rewritten D v = Avg where [ I V I v x, y, x, y,,, ÎV x, y,, ] 11 = ω I x +,y + ω I x +,y 1 Î V x, y,, = ω Î x +,y + ω Î x + 13,y. IV x, y,, represents the pixel value in the synthesize virtual view which is base on epth maps an uncompresse texture vieos, x +,y an x +,y are the pixel coorinates in the left an right views, respectively, an they are etermine by the epth maps, I x +,y an I x +,y are pixel values in the original left an right texture vieos, respectively; Î V x, y,, represents the pixel value at position x, y in the virtual view, x +,y an x +,y are the pixel coorinates in the left an right views, they are etermine by the compresse epth maps; Î x +,y an Î x +,y enote the pixel values in the compresse left an right texture vieos, respectively. Thereby, 14 can be erive from 11 D v = Avg [ω I Î ] +ω I Î 14 wherei, I, Î, an Î represent I x +,y, I x +,y, Î x +,y, an Î x +,y, respectively. Then, 15 can be erive as D v = ω Avg I Î +ω Avg I +ω ω Avg I Î I 15 where Avg I Î I Î approximates to zero, which can be prove in Appenix A. Thereby, 15 can be expresse by 16, where C 0 is a constant value which approximates to zero D v = ω Avg I Î + ω Avg I Î + C 0. The first item in 16 can be expane ω Avg I Î = ω Avg [ I 16 I + I Î ] [ = ω Avg ] I I [ ] +ω Avg I Î +ω [I Avg ] ] I [I Î 17 [ ] where Avg I I represents the mean square ifference cause by coorinate ifference which is inuce by epth compression, [ ] Avg I Î is the istortion of the [ left texture vieo cause by compression, an ] ] Avg I I [I Î can be prove to approximate to zero in Appenix B. Thereby, 17 can be expresse as 18 ω Avg I Î = ω [ D t + D0 ] + C1 18 where Dt enotes the istortion of the left texture vieo, D 0 represents Avg [ I I ], an C 1 approximates to zero. Similarly, the secon item of 16 can be represente as 19 ω Avg I = ω D t + D0 + C 19 where Dt enotes the istortion of the right view an D0 [ ] represents Avg I I. Then, 16 can be represente D v = ω D t + D0 + C 1 + ω D t + D0 + C + C0 = ω D t + ω D t + ω D0 + ω D 0 + C3 0 where ω D t + ω D t is inuce by the istortion of the left an the right texture vieos, ω D 0 + ω D 0 is inuce by the compression of epth maps, an C 3 approximates to zero. Equation 0 implies that D v can be ecompose to three items, the first item is a linear combination of texture istortiondt an Dt, the secon item is a linear combination ofd0 an D 0, an the thir item C 3 is a constant value which approximates to zero.

5 YUAN et al.: MODE-BASED JOINT BIT AOCATION BETWEEN TEXTUE VIDEOS AND DEPT MAPS FO 3-D VIDEO CODING 489 TABE I Distortion of the eft an the ight Texture Vieo epth value Z. Since there is a linear relationship between 1 / Z an isparity as shown in 1, 8 can be obtaine by replacing 1/ Z with/ fl = M/ fl + N. 8 Then, 9, 30 can be obtaine by taking 8 into 6 as D = M /fl Avg 9 B. Further Analysis on Virtual View s Distortion et π 1 enote ω Dt + ω D t an π enote ω D0 + ω D 0 for short. π 1 is influence by the istortion of both the left an right texture vieos. Table I shows the istortions of the left an the right view when both of them are compresse with the same QP. From Table I, we can conclue that Dt an Dt are similar when they are compresse by the same QP. That is because the left an the right texture vieos have a large amount of same contents, so π 1 can be approximate 1 ω + ω Dt 1 where D t represents the average istortion of the left an the right views. Ï is the istortion inuce by epth compression. Similar with the erivation of motion warping errors in [17], D0 an D0 can be characterize by a linear moel of Avg an Avg as shown in Appenix C D 0 = Avg D 0 = Avg 3 where, are two parameters which is etermine by the contents of the left an the right image, respectively [17]. As shown in Appenix C, an are the ifferences between original isparity, an the reconstructe isparity, = 4 =. 5 When epth maps are compresse, the istortion of epth maps D can be written D = Avg 6 where an are pixel values in the uncompresse epth maps an the compresse epth maps. Because epth values are quantize to the range 0 55 using non-uniform quantization shown in which can be simplifie to be as = M/ Z + N 7 where M an N are two constant parameters for fixe Z near an Z far, the istortion D can then be iffuse to the D = M /fl Avg 30 where D an D are the istortion of the left an right epth maps, respectively. Equations 9 an 30 inicate that the istortion of epth maps can be represente by the mean square ifference between the original isparities, an the reconstructe isparities,. Accoringly, 31 an 3 can be erive from 5, 9, an 30 D 0 = fl /M D 31 D 0 = fl /M D. 3 Thereby, ω D 0 + ω D 0 can be represente ω D0 + ω D 0 = fl / ω M D + ω D 33. Because the epth maps of the left an the right view have a large amount of similar contents, an are similar, D an D are also similar. Then, ω D 0 + ω D 0 can be rewritten ω D 0 + ω D 0 fl / M ω + ω D 34 where D is the average istortion of epth maps, an is the average value of an. C. Distortion Moel of Synthesize Virtual View Taking the inherent epth map inaccuracy cause by epth estimation algorithm or epth cameras an C 3 in 0 into account, the istortion of virtual view can be moele as D v = AD t + BD + C 4 35 where A, B, an C 4 are moel parameters. Equation 35 means that there exists a planar moel among D v, D t, an D, which will be verifie in Section V. Base on the erive istortion moel, optimal bit allocation algorithm is esigne in the next section.

6 490 IEEE TANSACTIONS ON CICUITS AND SYSTEMS FO VIDEO TECNOOGY, VO. 1, NO. 4, API 011 TABE II Correlation Coefficients Between Q an D Fig. 4. elationship between D an Q. IV. Moel-Base Optimal Bit Allocation Algorithm The mathematical expression of the optimal bit allocation problem can be escribe min D v t, 36 s.t. t + c where c is the total bits constraint. Because t,, D t, an D epen on the quantization of texture vieos an epth maps, the optimal bit allocation problem can be converte to the optimization problem of quantization step Q t an Q. Equation 36 can be represente min D v Q t,q Q t,q 37 s.t. Q t + Q c. Fig. 5. elationship between an 1/Q. TABE III Correlation Coefficients Between 1/Q an The D t -Q t, D -Q an t -Q t, -Q functions are analyze separately. A. D t -Q t an D -Q Moels D t -Q t an D -Q moels can be obtaine from the compression process of texture vieos an epth maps. For uniform quantization, the relationship between istortion an quantization step Q can be approximate to be Q /1 [18] when Q is small enough. In vieo coing applications, since Q is usually large, Q /1 is not accurate enough to represent the istortion. Thereby, linear D t -Q t moel [19] is use in this paper D t Q t = α t Q t + β t 38 where α t an β t are moel parameters which epen on coing structure an vieo contents. For epth maps, the relationship between D an Q is also foun as a linear function through experiments, as shown in Fig. 4 an Table II. Therefore, D - Q moel can be written D = α Q + β 39 where α an β are two parameters. Thus, D v can be written D v Q t,q = µq t + νq + C 40 where µ an ν represent the influence factors of Q t an Q on the virtual view s istortion, respectively, C is a constant value for certain texture vieos an epth maps. B. t -Q t an -Q Moels For texture vieos, a lot of t -Q t moels have been esigne for rate control, such as the well-known quaratic moel [0], fractional moel [1], an others. For easy implementation, fractional moel is use in this paper t = a t Q 1 t + b t 41 where a t an b t are moel parameters, an t is the bits consumption of the left an the right texture vieos. Experiments show that the fractional moel is also applicable for epth maps. The relationship between an 1/Q is shown in Fig. 5. The correlation coefficients between an 1/Q for ifferent epth maps are all larger than 0.96, as shown in Table III, which means that the fractional moel is accurate for epth maps. Therefore, the -Q moel can be written = a Q 1 + b 4 where a an b are moel parameters an is the bits consumption of the left an the right epth maps. C. Solution of the Bit Allocation Problem Base on the above analysis, t -Q t, -Q, an the propose D v Q t, Q moels are all efine by close-form expressions, the bit allocation problem can be rewritten as

7 YUAN et al.: MODE-BASED JOINT BIT AOCATION BETWEEN TEXTUE VIDEOS AND DEPT MAPS FO 3-D VIDEO CODING 491 min µq t + νq + C Q,Q t 43 s.t. a Q 1 + b + a t Q 1 t + b t c. Many optimization methos coul be use to fin the optimal solution of 43. owever, no matter what optimization metho is use, the optimal solution will be the same since there is only one optimal solution of 43. Because the agrangian multiplier metho can be applie easily, it is use to calculate the optimal solution of 43. The constraine optimization problem in 43 is mappe to an equivalent unconstraine optimization problem min F = µq t +νq +C+λ a t Q 1 t +b t +a Q 1 +b c Q t,q 44 where λ is the agrangian multiplier. Then, the optimal quantization steps Q t an Q can be calculate by solving the equations F = µ λa t Q t =0 Q t F Q a t Q 1 t = ν λa Q =0 + b t + a Q 1 + b c =0. Thereby, the optimal Q t an Q can be erive Q t = a t + νat a µ c b t b, Q = 45 µ a Q t. 46 ν a t D. Parameters Calculation In orer to calculate Q t an Q, the moel parameters µ, ν, a t, b t, a, an b must be firstly obtaine. Since, a t, b t, a, an b are parameters in t -Q t an - Q moels, they can be obtaine by pre-encoing the texture vieos an the epth maps with ifferent quantization steps, i.e., Q t,1, Q t,, Q,1, an Q,. Subsequently, t,1, t,,,1, an, can be obtaine, an parameters a t, b t, a, an b can be erive by solving equations t,1 = a t Q 1 t,1 + b t t, = a t Q 1 t, + b 47 t,1 = a Q 1,1 + b, = a Q 1, + b 48. µ an ν can also be obtaine in a similar manner. Equation 40 means that the virtual view s istortion varies linearly with the quantization steps of texture vieos an epth maps. Since we have assume that epth maps are accurate enough, the camera systems are well calibrate, an there are no color iscrepancies in all the acquire views, the virtual view which is synthesize from the uncompresse texture vieos an epth maps will be the same with the view in the synthesis position. Thus, the istortion of a virtual view which is synthesize from compresse texture vieos an epth maps can be calculate by comparing it with the one which is synthesize from the uncompresse texture vieos an epth maps. Therefore, in orer to calculate µ an ν, texture vieos an epth maps are first pre-encoe with quantization setups Q t,1, Q t,, Q,1, an Q,, respectively. Fig. 6. Flow chart of the propose bit allocation algorithm. Then, those compresse texture vieos an epth maps are use to synthesize three virtual views with ifferent quality at the same position. After that, the istortions of the three virtual views are calculate by comparing them with the one which is synthesize from the uncompresse texture vieos an epth maps. Finally, parameters µ an ν can be calculate by solving the equations D v,1 = µq t,1 + νq,1 + C D v, = µq t,1 + νq, + C D v,3 = µq t, + νq, + C 49 where D v,1 is the istortion of the synthesize virtual view when the quantization step of texture vieos is Q t,1 an the quantization step of epth maps is Q,1. D v, is the istortion of the synthesize virtual view when the quantization step of texture vieos is Q t,1 an the quantization step of epth maps is Q,. D v,3 is the istortion of the synthesize virtual view when the quantization step of texture vieos is Q t, an the quantization step of epth maps is Q,. The flow chart of the propose bit allocation algorithm is shown in Fig. 6. V. Experimental esults In this section, the istortion moel shown in 35 is verifie first, an then the performance an complexity of the propose bit allocation algorithm are compare with those of three bit allocation algorithms full search algorithm, iu s algorithm [7], an fixe bits ratio 5:1 bit allocation. In the experiments, five 3-DV sequences were use as shown in Table IV. Depth maps of Ballet was provie by Microsoft [], epth maps of Mobile, an Beergaren were provie by Philips [3], an the epth maps of ovebir1 an Bookarrival were generate by Depth Estimation Software [4] provie by MPEG-3-DV group..64/mvc reference software JMVC8.1 was use to encoe texture vieos an epth maps, an the configurations of the encoer are shown in Table V. View Synthesis eference Software [5] was use to synthesize virtual views. A. Distortion Moel Verification In this part, the istortion moel shown in 35 is verifie. Experiments were ivie into seven groups. In each group,

8 49 IEEE TANSACTIONS ON CICUITS AND SYSTEMS FO VIDEO TECNOOGY, VO. 1, NO. 4, API 011 TABE IV Sequences Use in the Experiments TABE V Configurations of Encoer TABE VI Configurations of Q an Q t Fig. 7. elationship among D v, D t, an D. a Mobile. b Beergaren. c Ballet. ovebir1. e Bookarrival. TABE VII Correlation Coefficients an Average Fitting Error Between the Actual D v an the Fitte D v Q were fixe, an Q t varie from 6.5 to 104 corresponing QP was from 0 to 44. The settings of Q an Q t in the experiment are shown in Table VI. In the experiment, texture vieos an epth maps were first compresse using those Q an Q t. Then, the istortion of texture vieos D t an epth maps D were calculate. Finally, virtual views were synthesize using those compresse texture vieos an epth maps, an the istortions of virtual views D v were calculate by comparing them with the view the central view in the synthesis position. Fig. 7 shows the D v surface which is etermine by D t an D. From Fig. 7, we can see that the D v surface is a planar surface which means that D v varies linearly with both D t an D. Therefore, the moel shown in 35 is accurate. D v is then fitte by D t an D through the planar moel in 35. Table VII shows the correlation coefficients between the fitte istortion an the actual istortion of virtual views. The average fitting error is also shown in Table VII. From Table VII, the correlation coefficients between the actual istortions an the fitte values are all larger than 0.97, meanwhile the average fitting errors are all small which means the erive moel is accurate. B. Bit Allocation Performance In this experiment, we compare the -D performance of the propose bit allocation algorithm with 3 bit allocation algorithms, i.e., full search algorithm [6], iu s algorithm [7], an fixe ratio 5:1 bit allocation metho. In the experiment, each sequence was teste in 5 bit rate constraints c, as shown in Table VIII. During the implementation of full search algorithm, the search range of QPs is set to be [0, 45]. The -D curves of ifferent bit allocation algorithms are shown in Fig. 8. The vertical axis in Fig. 8 enotes virtual views peak-signal-to-noise-ratio which were obtaine by comparing them with the view in the synthesis position, an the horizontal axis shows the bit rate constraints. Because of the preiction errors of the parameters in 47 49, the propose algorithm coul not give the same -D performance as that of full search algorithm. owever, it can be conclue from Fig. 8 that the performance of the propose algorithm is close to those search-base algorithms. Moreover, compare with the fixe ratio 5:1 bits allocation metho, a maximum 1. B gain Ballet, c =1.5Mb/s coul be obtaine by the propose algorithm. C. Complexity Analysis The complexity of joint bit allocation algorithms mainly epens on the pre-encoing process an virtual view synthesis.

9 YUAN et al.: MODE-BASED JOINT BIT AOCATION BETWEEN TEXTUE VIDEOS AND DEPT MAPS FO 3-D VIDEO CODING 493 TABE VIII Bit ate Constraints for Each Sequence Therefore, the time consuming of the pre-encoing process in the propose algorithm is only about 7.69% of that of full search algorithm; meanwhile the time consuming of the view synthesis in the propose algorithm is about 0.6% of that of full search algorithm. The complexity comparisons among full search algorithm, iu algorithm, an the propose algorithm are shown in Fig. 9 wherein we can see that compare with full search an iu s algorithms, the complexity of the propose algorithm is very low. Fig. 8. D curves of ifferent bit allocation algorithms. a Mobile. b Beergaren. c Ballet. ovebir1. e Bookarrival. VI. Conclusion This paper presente a moel-base joint bit allocation algorithm for 3-D vieo coing. We erive a planar istortion moel of synthesize virtual views analytically. Base on the erive moel, the bit allocation problem was formulate as a constraine optimization problem, an was solve by agrangian multiplier metho. Since the optimal QPs of texture vieos an epth maps can be calculate numerically, it is no nee to search the optimal QPs. Full search algorithm, iu s algorithm [7], an fixe ratio 5:1 bit allocation metho were use as the benchmarks to emonstrate the performance of the propose algorithm. Experimental results showe that the erive istortion moel is accurate, an the -D performance of the propose algorithm is close to those search-base algorithms, i.e., full search algorithm, iu s algorithm, while the complexity of the propose algorithm is low. Besies, compare with the fixe ratio 5:1 bits allocation metho, a maximum 1. B gain can be achieve by the propose algorithm. Moreover, the propose istortion moel of synthesize virtual views coul also be use in moe ecision of 3-D vieo coing because the compression of texture vieos an epth maps shoul also be oriente to the quality of synthesize virtual views. Our future work is to esign an -D criterion for the moe ecision of 3-D vieo coing by employing the propose moel. Fig. 9. Complexity comparison. In full search algorithm, the complexity varies with the QP search range. Since the QP search range in the experiments is 0 45, both texture vieos an epth maps shoul be preencoe for 6 times separately, an virtual views shoul be synthesize 6 6 times. In iu s algorithm [7], it is no nee to search all the QPs an synthesize virtual views so many times, an the complexity of iu s algorithm is about one fourth of that of full search algorithm [7]. The propose algorithm oes not nee to search. Both texture vieos an epth maps shoul only be pre-encoe for twice, meanwhile virtual views shoul only be synthesize four times. Appenix A emma 1: Avg I Î I approximates to zero. Proof: As inicate by the aw of arge Numbers, when sample numbers are large, the average value of all the samples approximates to their expectation; emma 1 can be represente E I Î I 0 50 where E enotes the expectation operation. I Î an I Î can be written I Î = I Î + Î Î 51 = e t + Î Î

10 494 IEEE TANSACTIONS ON CICUITS AND SYSTEMS FO VIDEO TECNOOGY, VO. 1, NO. 4, API 011 I = I + Î 5 = e t + Î where Î an Î represent the pixel values at position x +,y an x +,y in the compresse left an right texture vieos, respectively, e t an e t are the quantization errors of the left an the right texture vieos. Then, 53 can be obtaine E I Î I [ = E e t +Î ] [ Î e t +Î ] ] + E [Î Î e t = E e t e t ] + E e t [Î Î ] + E [Î Î [Î Î ]. 53 Since there are a lot of common contents between the left an the right texture vieos, an there exists a horizontal isplacement between the left an the right texture vieos, e t an e t are also similar, an there also exists a horizontal isplacement between e t an e t. E e t e t can be represente E e t e t E e t e t 54 where is the isplacement between x an x. Equation 54 means that E e t e t approximates to the auto correlation function of e t. Since, the quantization noise resulting from uniform scalar quantization can be moele as a white noise moel [6], thereby, e t can be assume to be a white noise. Accoringly, the auto correlation function of e t approximates to zero, that is E e t e t approximates to zero when oes not equal to zero. In the secon item, Î is the pixel ifference cause by coorinate ifference which is inuce by epth compression, an Î can be represente Î = Î From the efinition of iscrete Fourier transform DFT, the DFT of Î x, y are shown X u, v = DFT Î x, y x=0 Then, Î y=0 Î x, y exp jπ ux W + vy an Î Î u=0 v=0 exp jπ Î u=0. 56 can be represente X u, v exp j π W u ux W + vy v=0 exp jπ ux W + vy 57 X u, v exp j πw u 58. et K x, y, u, v enotes X u, v exp jπ ux, u, enotes exp j π W u, Ɣ u, enotes exp j π W u, an then Î can be represente Î u=0 v=0 K x, y, u, v u, 59 1 Ɣ u,. W + vy Ɣ u, can be expane via Taylor series at =0as Ɣ u, =1+j π W u E e t ] [Î = e 1 E t u=0 v=0 E e t u=0 v=0 K x, y, u, v u, j π W u 1 u, E j π W u e t K x, y, u, v 1 e t K x, y, u, v 6 ] [Î Î u, E j π W ue t K x, y, u, v E 1 E e t K x, y, u, v 63 u=0 v=0 ] ] E [Î Î [Î 1 E v 1 =0 v =0 u 1 =0 u =0 1 = u1, v 1 =0 v =0 u 1 =0 u =0 K x, y, u 1,v 1 u1, K x, y, u,v u, u, K x, y, u 1,v 1 K x, y, u,v [ ] π u 1 u W [ ] π u 1 u E W 68

11 YUAN et al.: MODE-BASED JOINT BIT AOCATION BETWEEN TEXTUE VIDEOS AND DEPT MAPS FO 3-D VIDEO CODING 495 where 1 is the high-orer infinitesimal about.so 59 can be written Î K x, y, u, v u, u=0 v=0 j π W u Therefore, the secon item of 53 can be represente as 6, as shown at the bottom of the previous page. Because epens on the compresse epth maps, an e t K x, y, u, v is cause by the compresse texture vieos, an e t K x, y, u, v are inepenent. Then 6 can be represente as 63, as shown at the bottom of the previous page. In 63, E can be erive as follows. Because epth values are quantize to the range 0 55 using non-uniform quantization shown in, which is simplifie = M/ Z + N 64 where is the pixel value in the uncompresse epth maps, Z enotes the epth value, an M an N are two constant parameters for a certain Z near an Z far. Since there exists a linear relationship between 1/ Z an isparity as shown in 1, 65 can be obtaine by replacing 1/ Z with/ fl = M/ fl + N. 65 I I u=0 v=0 X u, v exp j π ux W u exp jπ W + vy 1 exp j π W u 76 I Î u=0 v=0 [ X u, v X ] ux u, v exp jπ W + vy exp j π W u + 77 [ E I = 1 E I ] [I Î ] v 1 =0 v =0 u 1 =0 u =0 [Ɣ u, Ɣ u 1 + u, ] T u 1,v 1 T u,v X u 1,v 1 δ x u,v u1 + u, 78 [ E I 1 E ][ ] Î Î Î v 1 =0 v =0 u 1 =0 u =0 T u,v T u,v X u,v δ x u,v u1 + u, [ D0 = Avg ] I I 1 = E u=0 v=0 X u, v u, [ ] = E I I T u, v 1 exp j π W u j π W u S u, v X u, v u, T u, v 1 exp j π W u 83 u=0 v=0 S u, v u=0 v=0 X u, v u, T u, v j π W u 1 1 u=0 v=0 X u, v u, T u, v j π W u 84 1 D0 π E X u 1,v 1 X u,v u1 u, T u1 u,v 1 v u 1 u. W u 1 =0 v 1 =0 u =0 v =0 85 = 1 E u 1 =0 v 1 =0 u =0 v =0 X u 1,v 1 X u,v u1 u, π T u1 u,v 1 v u 1 u 87 W

12 496 IEEE TANSACTIONS ON CICUITS AND SYSTEMS FO VIDEO TECNOOGY, VO. 1, NO. 4, API 011 Accoringly, the quantization errors of the left an the right epth maps can be written e = M fl e = M fl = M fl = M fl Since, the quantization error can be moele as a zero mean white noise [6], the expectations of e t, e t, e, an e are zeros. Thereby, E e, E e, E, an E approximate to zero. Thus, the secon item of 53 approximates to zero. The thir item of 53 coul also be prove to approximate to zero in a similar way. Base on the above analysis, the fourth item in 53 can be represente as 68. Similar with the analysis of the first item E e t e t, E can also approximate to zero, thereby, ] ] E [Î Î [Î approximates to zero. Thus, 50 coul be prove. Î u=0 v=0 exp jπ ux W + vy X u, v exp j πw u. 75 Thereby, 76 an 77 coul be erive, which are shown at the bottom of the previous page. et T u, v, u,, Ɣ u,, δ x u, v enote exp jπ ux W + vy,exp j π W u, exp j π W u, an X u, v X u, v, respectively, then 78 can be erive, as shown at the bottom of the previous page. Equation 78 can be represente as 79 when Ɣ u, an Ɣ u 1 + u, is expane via Taylor series. Because the expectation of approximates to zero as shown in proof of emma 1, 79 can be rewritten [ E I Î ][ ] Î Î [ emma : Avg ] [I Î Appenix B ] I I approximates to zero. Proof: Similar with the proof[ of emma 1, emma ] can be represente as E I I ] [I Î, which approximates to zero. The DFT of I x, y,i, I, an Î can be expresse X u, v = DFT I x, y x=0 y=0 I x, y exp jπ ux W + vy X1 u, v = DFT I = X u, v exp 69 j π W u 70 X1 u, v = DFT I = X u, v exp j πw u 71 X Î u, v = DFT = X u, v exp j πw u. 7 Then, I, I, an Î can be represente I u=0 v=0 X u, v exp j π W u exp jπ ux W + vy 73 I u=0 v=0 X u, v exp j πw u exp jπ 74 ux W + vy Appenix C emma 3: D0 an D 0 can be characterize by a linear moel about Avg an Avg. Proof: Taking D0 as an example, accoring to Appenixes A an B, D0 can be represente as 81 which is shown at the bottom of the previous page. Equation 81 can then be represente D 0 = E S u, v = E S u, v S u, v 8 where S u, v is shown in 83 which is given below, an S u, v is the conjugate of S u, v. Exp j π W u can be expane via Taylor series, an S u, v can be represente as 84, then D0 can be represente as 85. Equations 84 an 85 are shown at the bottom of the previous page. Therefore, 86 can be obtaine D0 E 86 where can be represente as 87. Similarly, D0 represente can be D 0 E. 88 Thus, from the aw of arge Numbers, D 0 an D 0 can be characterize by a linear moel about Avg an Avg. Acknowlegment The authors woul like to thank M. i for his great help on this paper an the eitors an anonymous reviewers for their valuable comments. They woul also like to thank Microsoft esearch, Beijing, China, Fraunhofer I, Berlin, Germany, GIST, Singapore, ETI, Daejeon, Korea, Nagoya University, Nagoya, Japan, an Poznan University, Poznan, Polan, for proviing their 3-D vieo sequences an their valuable work on 3-DV.

13 YUAN et al.: MODE-BASED JOINT BIT AOCATION BETWEEN TEXTUE VIDEOS AND DEPT MAPS FO 3-D VIDEO CODING 497 eferences [1]. Onural, A. Gotchev,. M. Ozaktas, an E. Stoykova, A survey of signal processing problems an tools in holographic 3-D television, IEEE Trans. Circuits Syst. Vieo Technol., vol. 17, no. 11, pp , Nov [] M. Tanimoto an T. Fujii, FTV: Free viewpoint television, in Proc. 61th Meet. ISO/IEC JTC1/SC9/WG11, Jul. 00, no. M8595. [3] A. A. Alatan, Y. Yemez, U. Güükbay, X. Zabulis, K. Müller, Ç. E. Erem, C. Weigel, an A. Smolic, Scene representation technologies for 3-DTV: A survey, IEEE Trans. Circuits Syst. Vieo Technol., vol. 17, no. 11, pp , Nov [4] C. Fehn, Depth-image-base renering DIB, compression an transmission for a new approach on 3-D-TV, Proc. SPIE, Stereoscopic Image Process. ener., vol. 591, pp , Jan [5] Telecommunication Stanarization Sector of ITU, Series : Auio Visual an Multimeia Systems Infrastructure of Auiovisual Services: Coing of Moving Vieo, Annex, ecomm. ITU-T.64, Mar [6] Y. Morvan, D. Farin, an P.. N. e With, Joint epth/texture bit allocation for multi-view vieo compression, in Proc. 6th PCS, Nov. 007, pp [7] Y. iu, Q. uang, S. Ma, D. Zhao, an W. Gao, Joint vieo/epth rate allocation for 3-D vieo coing base on view synthesis istortion moel, Signal Process.: Image Commun., vol. 4, no. 8, pp , Sep [8] N. K. Kanhere an S. T. Birchfiel, A taxonomy an analysis of camera calibration methos for traffic monitoring applications, IEEE Trans. Intell. Trans. Syst., vol. 11, no., pp , Jun [9] C. Doutre an P. Nasiopoulos, Color correction preprocessing for multiview vieo coing, IEEE Trans. Circuits Syst. Vieo Technol., vol. 19, no. 9, pp , Sep [10] M. Kawakita, Axi-vision camera: eal-time epth-mapping camera, in Proc. 8th Meet. ISO/IEC JTC1/SC9/WG11, ocument M14949, Oct [11] M. Tanimoto, T. Fujii, an K. Suzuki, Multi-view epth map of ena an Akko&Kayo, in Proc. 8th Meet. ISO/IEC JTC1/SC9/WG11, ocument M14888, Oct [1] M. Tanimoto, T. Fujii, an K. Suzuki, Improvement of epth map estimation an view synthesis, in Proc. 83th Meet. ISO/IEC JTC1/SC9/WG11, ocument M15090, Jan [13] M. Tanimoto, T. Fujii, an K. Suzuki, Experiment of view synthesis using multi-view epth, in Proc. 8th Meet. ISO/IEC JTC1/SC9/WG11, ocument M14889, Oct [14] I. Felmann, M. Mueller, F. Zilly,. Tanger, K. Mueller, A. Smolic, P. Kauff, an T. Wiegan, I test material for 3-D vieo, in Proc. 84th Meet. ISO/IEC JTC1/SC9/WG11, ocument M15413, Apr [15] P. Benzie, J. Watson, P. Surman, I. akkolainen, K. opf,. Urey, V. Sainov, an C. von Kopylow, A survey of 3-DTV isplays: Techniques an technologies, IEEE Trans. Circuits Syst. Vieo Technol., vol. 17, no. 11, pp , Nov [16]. Yuan, Y. Chang, M. i, an F. Yang, Moel base bit allocation between texture images an epth maps, in Proc. Int. Conf. CCTAE, vol. 3. Aug. 010, pp [17]. Mathew an D. S. Taubman, Qua-tree motion moeling with leaf merging, IEEE Trans. Circuits Syst. Vieo Technol., vol. 0, no. 10, pp , Oct [18]. M. Gray an D.. Neuhoff, Quantization, IEEE Trans. Inform. Theory, vol. 44, no. 6, pp , Oct [19]. Wang an S. Kwong, ate-istortion optimization of rate control for.64 with aaptive initial quantization parameter etermination, IEEE Trans. Circuits Syst. Vieo Technol., vol. 18, no. 1, pp , Jan [0]. J. ee, T. Chiang, an Y. Q. Zhang, Scalable rate control for MPEG- 4 vieo, IEEE Trans. Circuits Syst. Vieo Technol., vol. 10, no. 6, pp , Sep [1] S. Ma, W. Gao, an Y. u, ate-istortion analysis for.64/avc vieo coing an its application to rate control, IEEE Trans. Circuits Syst. Vieo Technol., vol. 15, no. 1, pp , Dec [] C.. Zitnick, S. B. Kang, M. Uyttenaele, S. Winer, an. Szeliski, igh-quality vieo view interpolation using a layere representation, ACM SIGGAP ACM Trans. Graphics, vol. 3, no. 3, pp , Aug [3] Philips 3-DV. Depth Maps of Mobile an Beergaren [Online]. Available: FTP://mpeg ph39@ftp.ehv.campus.philips.com [4] MPEG-3-DV Depth Estimation eference Software [Online]. Available: estimation [5] MPEG-3-DV View Synthesis eference Software [Online]. Available: synthesis [6]. Xiao, M. Johansson,. ini, S. Boy, an A. Golsmith, Joint optimization of communication rates an linear systems, IEEE Trans. Automat. Control, vol. 48, no. 1, pp , Jan ui Yuan S 08 receive the B.E. an Ph.D. egree in telecommunication engineering from Xiian University, Xi an, China, in 006 an 011, respectively. e is currently a ecturer with the School of Information Science an Engineering, Shanong University, Jinan, China. e is also a esearcher with the State Key aboratory of Integrate Services Networks, Xiian University, Xi an, an isense State Key aboratory of Digital Multi-Meia Technology Company, t., Qingao, China. is current research interests inclue vieo coing, multimeia communication, an image processing. Yilin Chang receive the B.S. egree from Xi an Jiaotong University, Xi an, China, in 1970 an the M.S. egree in communication an information systems from Xiian University, Xi an, in 198. e is currently a Professor with the State Key aboratory of Integrate Services Networks, Xiian University. is current research interests inclue multimeia in networks, vieo compression, vieo transmission, an network management. Junyan uo receive the B.S., M.S., an Ph.D. egrees in communication an information systems from Xiian University, Xi an, China, in 003, 006, an 008, respectively. She is currently with the State Key aboratory of Integrate Services Networks, Xiian University. In 008 an 010, she was a ecturer an an Associate Professor with Xiian University. er current research interests inclue multiview vieo coing, 3-D vieo coing, an vieo pre-processing. Fuzheng Yang M 10 receive the B.E. egree in telecommunication engineering, an the M.E. an Ph.D. egrees in communication an information systems from Xiian University, Xi an, China, in 000, 003, an 005, respectively. e is currently with the State Key aboratory of Integrate Services Networks, Xiian University. In 005 an 006, he was a ecturer an an Associate Professor with Xiian University. From 006 to 007, he was a Visiting Scholar an a Post-Doctoral esearcher with the Department of Electronic Engineering in Queen Mary University of onon, onon, U.K. is current research interests inclue vieo quality assessment, vieo coing, an multimeia communication. Zhaoyang u receive the B.S., M.S., an Ph.D. egrees in communication an information systems from Xiian University, in 198, 1985, an 1990, respectively. e is currently a Professor with the School of Telecommunications Engineering, Xiian University, Xi an, China. is current research interests inclue image processing an pattern recognition.