INTRINSIC DECOMPOSITION FOR STEREOSCOPIC IMAGES

Size: px

Start display at page:

Download "INTRINSIC DECOMPOSITION FOR STEREOSCOPIC IMAGES"

Joan Juliana Booker
6 years ago
Views:

1 INTRINSIC DECOMPOSITION FOR STEREOSCOPIC IMAGES Dehua Xie 1, Shuaicheng Liu 1, Kaio Lin 2, Shuyuan Zhu 1, and Bing Zeng 1 1 School of Electronic Engineering, University of Electronic Science and Technology of China 2 Dept. of Electrical & Coputer Engineering, National University of Singapore ABSTRACT Intrinsic iage decoposition is an iportant technique that decoposes an iage into reflectance and shading coponents. In this paper, we enable intrinsic decoposition for stereoscopic iages. Traditional approaches cannot be directly applied to decopose stereoscopic iages, yielding inconsistent reflectance and 3D artifacts after recoloring. To solve this proble, we propose a straight yet effective ethod for stereoscopic intrinsic decoposition, which consists of classical retinex constraint as well as disparity constraint. The forer encodes the shading soothness prior while the latter controls the reflectance siilarity between two views. To further reduce abiguity, we eploy local and non-local texture cues by using superpixels within and across two views. The experients show that our ethod can effectively decopose stereoscopic iages with high quality and offer a cofortable 3D viewing experience. Index Ters Stereoscopic, intrinsic iage decoposition, retinex, disparity 1. INTRODUCTION With the success of 3D ovies, stereoscopic caeras and displays becoe ore and ore popular. In recent years, various ethods have been proposed to target stereoscopic iages/videos processing. To nae a few, stereoscopic warping [1], cloning [2],inpainting [3], panoraa [4],retargeting [5], stabilization [6] are all popular iage/video editing research topics. However, aong all these studies, fewer work has been proposed to address the intrinsic decoposition proble for stereoscopic iages, which is one of the ost classical and iportant topics in iage editing. Intrinsic iage decoposition has decades of history in vision and graphics counities [7, 8, 9]. It ais at decoposing an iage I into the reflectance (R) coponent and the shading (S) coponent as follows: I = R S, (1) where denotes the pixel-to-pixel ultiplication. Many existing probles can benefit fro such a decoposition with the presence of these useful confounded inforation. For exaples, resurfacing and aterial recognition prefer a reflectance iage without appearance variations caused by shading [10]; and the shape-fro-shading algorith will be able to infer the surface geoetry ore accurately when less reflectance variation appears in the shading ap [11]. Though there continuously are a nuber of reported works contributing to this field, e.g., [12, 13, 14], intrinsic decoposition still reains a challenging proble. It is highly ill-posed as the nuber of unknowns (R, S) is twice as any as the observations (I). Furtherore, these traditional onocular decopositions cannot be directly applied to stereoscopic iages, due to the resulting inconsistent reflectance. By considering several extra constraints within and across views, our approach can deliver ore pleasant results. Fig. 1 shows the effect of recoloring. 2. RELATED WORK Intrinsic decoposition is first introduced in [7], which proposed the retinex algorith to analyze local iage deviations in shading and reflectance. Soe refined retinex algoriths with different assuptions were reported in [14, 15, 16]. To further reduce the abiguity, soe ethods [12, 17] tried to learn the priors to judge the iage derivatives, while others [14, 18] proposed additional constraints to reduce the nuber of unknowns. Specifically, Weiss et. al. [15] used ultiple iages to regularize the syste. However, it requires an accurate iage registration, which is challenging in any scenarios. Kang et. al. [19] proposed a ethod for stereoscopic decoposition. However, their ethod ainly focuses on graphics iages and is only conducted on synthetic data. In this paper, we propose a ore general approach that can be applied to stereo pairs captured by consuer-level caeras. Stereoscopic iage/video processing is an iportant topic, which has attracted lots of attention in the counity. Consistent anipulation on two different views is crucial for high quality stereo editing. To this end, Wang et. al. [20] explored an inpainting technique for stereoscopic iages, where both color and depth can be recovered siultaneously. Iage cloning ethods [2] copies the content and pastes it into a new 3D environent under the stereoscopic 3D disparity constraints. Meanwhile, iage warping is one of the ost basic anipulates in iage editing. For instance, Niu et. al. [1] extended the conventional 2D iage warping to

Fig. 1. An illustration of recoloring on stereoscopic iages. handle 3D stereoscopic iages; Lang et. al. [21] exploited a locally adaptive algorith for nonlinear disparity apping; Du et. al. [22] proposed to change the perspective for stereo pictures; and Didyk et.

2 Fig. 1. An illustration of recoloring on stereoscopic iages. handle 3D stereoscopic iages; Lang et. al. [21] exploited a locally adaptive algorith for nonlinear disparity apping; Du et. al. [22] proposed to change the perspective for stereo pictures; and Didyk et. al. [23] introduced a perceptual odel of disparity. More recently, stereo iage cropping [24] and authoring [25] have also been explored. 3. OUR METHOD Take the logarith at both sides of Eq. (1) gives, I = R + S. For siplicity, we reuse I, R, and S to represent their log values. We decopose stereoscopic iage pairs by optiizing over the following energy with respect to shading S: arg in E(S) =λ r E r (S)+λ d E d (S) S + λ l E l (S)+λ nl E nl (S)+λ a E a (S), (2) where E r, E nl, E d, E l, and E a are the retinex ter, disparity ter, local ter, non-local ter and absolute scale ter, respectively, with λ s, λ r, λ l, λ nl and λ a being the associated weights. Each of these ters will be elaborated in detail in the following. Notice that the reflectance R can be obtained by I S. Through experients, we set λ s = λ r = λ l = λ nl =1 and λ a = 1000 for all exaples in our syste Retinex Constraint According to the retinex theory, large derivatives in iage intensity attribute to reflectance changes, while sall derivatives are given to shading variations. Thus, the retinex ter E r (S) is forulated as a weighted su of shading and reflectance of neighboring pixels over the whole iage region [18]: E r (S)= [ (S S n ) 2 +ω,n (R R n ) 2], (3) n N () where stands for all pixels and N () denotes the four adjacent neighbors of. Substituting R by I S, yields: E r (S) = [ (1 + ω,n )ΔS,n 2 +ΔI,n 2 n N () 2ω,n ΔI,n ΔS,n ], (4) where ΔS,n = S S n, ΔI,n = I I n, and ω,n is a balancing weight that can be calculated in a nuber of ways [7, 18]. Here, we evaluate the chroaticity distance of neighboring pixels. If the chroaticity distance is greater than a threshold τ 1, we set ω,n =0, otherwise it is set to 100. Large reflectance variations are allowed, only when its chroaticity varies draatically. The chroaticity distance is defined as: ( ), n D(, n) =2 1 (5) 2 n 2 where and n denote color values of pixels and n Disparity Constraint Per-pixel dense disparity estiation ethods are well docuented in [26]. In our ipleentation, we calculate dense optical flow [27] between two views for disparity correspondances. We drop the inaccurate flow (e.g., occluded pixels) by checking the pixel color siilarity following the flow. Other estiation ethods can also be adopted [28]. We want to encourage the corresponding pixels between two views to have siilar reflectance. Given the per-pixel correspondences, we define the disparity ter E d (S) as: E d (S) = (R l R r )2, (6) where R r and R l refer to the left and right reflectance 2, and are atched disparity point pair. By substituting R with I S, Eq. (6) can be rewritten as: E d (S) = (I l I r Sl + S r )2. (7) 3.3. Local Constraint Local texture cues have shown their effectiveness to reduce the abiguity [18]. We eploy superpixels [29] to represent iage regions with siilar color and textures. We encourage the local siilarity within the superpixels on reflectance, which gives rise to the local ter E l (S): E l (S) = (I I n S + S n ) 2, (8) Pl i Γ l,n Pl i 1 τ is set to in our ipleentation. 2 If no l and r are specified, we represent both.

Fig. 2. An illustration of local constraints where P i l represents a superpixel group, Γ l denotes the set of superpixel groups, and and n are two different pixels fro the sae superpixel.

The nuber of sapled points is deterined by the size of a superpixel. Notably, local constraints are applied to two views separately. 3.4.

3 Fig. 2. An illustration of local constraints where P i l represents a superpixel group, Γ l denotes the set of superpixel groups, and and n are two different pixels fro the sae superpixel. An illustration of superpixels for local constraint is shown in Fig. 2. We saple several pairs of points (yellow dots highlighted in Fig. 2 (b), (c)) in each superpixel (Fig. 2 (d)). The nuber of sapled points is deterined by the size of a superpixel. Notably, local constraints are applied to two views separately Non-Local Constraint Figure 3 shows an exaple of non-local constraints. We encourage superpixels with siilar iage color to share a siilar reflectance, which can effectively reduce the abiguity of the syste. Particularly, for each superpixel, we find the K nearest neighbors in ters of iage color in both views and saple several pairs of locations inside the as illustrated in Fig. 3. We can easily spot that there are any siilar nonadjacent patches within and across two views. The non-local ter E nl (S) is then defined as: E nl (S) = (I I n S +S n ) 2, (9) Pnl i,p j nl Γ nl Pnl i,n P j nl where Γ nl is the union of all superpixels fro both views. Copared to the local constraints, the difference between Eq. (9) and Eq. (8) is that pixels and n are not in the sae superpixel but different superpixels Absolute Scale Constraint Finally, we add the absolute scale ter as follows: E a (S) = (S 1) 2, (10) P a where the set P a denotes two brightest pixels, one for each view. We constrain the brightest pixels in two views to have a unit shading, which further regularizes the syste. It is iportant to point out that all ters in Eq. (2) are quadratic, yielding a sparse linear syste which is iniized by the Matlab \ solver. Fig. 3. An illustration of non-local constraints #1 #2 #3 #4 #5 without with Table 1. The APD values of exaples in Fig. 4 under with/out disparity constraints 4. EXPERIMENTS Aong all the five ters in Eq. (2), the retinex, local and absolute are quite classical. Here, we would like to validate the effectiveness of disparity and non-local ters, both visually and nuerically, following which we present our final reuslts in Fig With and Without Disparity Constraint To evaluate the effect of the disparity ter on reflectance consistency nuerically, we define the average pixel distance (APD) as follows: AP D(R l,r r )=, Φ R l R r 2 N(Φ), (11) where Φ is the set of all atched pixels in two views and N(Φ) is the total nuber of atched pixel pairs. We copare the color difference of corresponding reflectance nuerically (note that the color is noralized to [0, 1]). Table 1 lists APD values with and without the disparity constraints on the exaples in Fig. 4. Without the disparity constraints, the reflectance is inconsistent between two views, which is indicated by the larger APD values. Moreover, we use the ethod of [30] to decopose our 5th exaple in Fig. 4. As it is a onocular ethod, it is applied to two views separately. The reflectances are shown in Fig. 5, which is inconsistent between two views. Notably, we adopt the sae paraeters for all exaples.

4 #1 #2 #3 #4 #5 Left Iage Right Iage Left Shading Right Shading Left Reflectance Right Reflectance Fig. 4. Our results. More results are provided in the suppleentary. Iages are captured by a steroscopic caera of ourselves. Fig. 5. The reﬂectances produced by ethod [30] on Fig. 4 #5. The inconsistence is highlighted by the red arrows With and Without Non-Local Constraint Figure 6 shows a visual coparison of with and without the non-local constraints. Here, we conduct the experient on a single iage. The input iage (Fig. 6(a)) is segented by superpixles (Fig. 6(b)). Notice the color difference in the reﬂectance iage (Fig. 6(c)) as highlighted by the red arrow. Our result (Fig. 6(d)) is ore consistent. Fig. 6. With and without non-local constraints. Iage credits: is licensed under Creative Coons. consistent reﬂectance decoposition. Our ethod has been validated on various stereoscopic iages to produce a better perforance as copared to the conventional approaches that are developed for onocular iages. 5. CONCLUSION 6. ACKNOWLEDGE We have presented a ethod for intrinsically decoposing stereoscopic iages. Our ethod cobines several constraints, naely retinex, disparity, local, non-local, and absolute scale constraints, into a uniﬁed fraework to produce a We would like to thank all reviewers for their coents to iprove our paper. This work has been supported by National Natural Science Foundation of China ( , and )

5 References [1] Y. Niu, W. Feng, and F. Liu, Enabling warping on stereoscopic iages, ACM TOG, vol. 31, no. 6, pp. 183, [2] S. Luo, I. Shen, B. Chen, W. Cheng, and Y. Chuang, Perspective-aware warping for sealess stereoscopic iage cloning, ACM TOG, vol. 31, no. 6, pp. 182:1 182:8, [3] T. Mu, J. Wang, S. Du, and S. Hu, Stereoscopic iage copletion and depth recovery, The Visual Coputer, vol. 30, no. 6-8, pp , [4] F. Zhang and F. Liu, Casual stereoscopic panoraa stitching, in CVPR, [5] K. Lee, C. Chung, and Y. Chuang, Scene warping: Layer-based stereoscopic iage resizing, in CVPR, 2012, pp [6] F. Liu, Y. Niu, and H. Jin, Joint subspace stabilization for stereoscopic video, in ICCV, 2013, pp [7] E. Land and J. McCann, Lightness and retinex theory, Journal of the Optical Society of Aerica A,, vol. 61, no. 1, pp. 1 11, [8] H. Barrow and J. Tenenbau, Recovering intrinsic scene characteristics fro iages, Coputer Vision Systes, p. 2, [9] Christophe Schlick, A survey of shading and reflectance odels, in Coputer Graphics Foru, 1994, vol. 13, pp [10] G. Ye, E. Garces, Y. Liu, Q. Dai, and D. Gutierrez, Intrinsic video and applications, ACM TOG, vol. 33, no. 4, pp. 80, [11] R. Zhang, P.-S. Tsai, J. Cryer, and M. Shah, Shape fro shading: a survey, IEEE TPAMI, vol. 21, no. 8, pp , [12] T. Narihira, M. Maire, and S. Yu, Learning lightness fro huan judgeent on relative reflectance, in CVPR, 2015, pp [13] S. Bi, X. Han, and Y. Yu, An l 1 iage transfor for edge-preserving soothing and scene-level intrinsic decoposition, ACM TOG, vol. 34, no. 4, pp. 78, [14] L. Shen, P. Tan, and S. Lin, Intrinsic iage decoposition with non-local texture cues, in CVPR, 2008, pp [15] Y. Weiss, Deriving intrinsic iages fro iage sequences, in ICCV, 2001, vol. 2, pp [16] A. Bousseau, S. Paris, and F. Durand, User-assisted intrinsic iages, in ACM TOG, 2009, vol. 28, p [17] J. Barron and J. Malik, Shape, illuination, and reflectance fro shading, IEEE TPAMI, vol. 37, no. 8, pp , [18] Q. Zhao, P. Tan, Q. Dai, L. Shen, E. Wu, and S. Lin, A closed-for solution to retinex with nonlocal texture constraints, IEEE TPAMI, vol. 34, no. 7, pp , [19] J. Kang, B. Jiang, J. Chen, and X. Liu, Stereoscopic iage recoloring via consistent intrinsic decoposition, in ICVRV. IEEE, 2014, pp [20] L. Wang, H. Jin, R. Yang, and M. Gong, Stereoscopic inpainting: Joint color and depth copletion fro stereo iages, in CVPR, 2008, pp [21] M. Lang, A. Hornung, O. Wang, S. Poulakos, A. Solic, and M. Gross, Nonlinear disparity apping for stereoscopic 3d, ACM TOG, vol. 29, no. 4, pp. 75, [22] S. Du, S. Hu, and R. Martin, Changing perspective in stereoscopic iages, IEEE TVCG, vol. 19, no. 8, pp , [23] P. Didyk, T. Ritschel, E. Eiseann, K. Myszkowski, and H. Seidel, A perceptual odel for disparity, ACM TOG, vol. 30, no. 4, pp. 96, [24] F. Zhang, Y. Niu, and F. Liu, Making stereo photo cropping easy, in ICME, 2013, pp [25] F. Liu, Y. Niu, and H. Jin, Casual stereoscopic photo authoring, IEEE TMM, vol. 15, no. 1, pp , [26] D. Scharstein and R. Szeliski, A taxonoy and evaluation of dense two-frae stereo correspondence algoriths, IJCV, vol. 47, no. 1-3, pp. 7 42, [27] C. Liu, Beyond pixels: exploring new representations and applications for otion analysis, Ph.D. thesis, MIT, [28] H. Hirschüller and D. Scharstein, Evaluation of stereo atching costs on iages with radioetric differences, IEEE TPAMI, vol. 31, no. 9, pp , [29] P. Felzenszwalb and D. Huttenlocher, Efficient graphbased iage segentation, IJCV, vol. 59, no. 2, pp , [30] S. Bell, K. Bala, and N. Snavely, Intrinsic iages in the wild, ACM TOG, vol. 33, no. 4, pp. 159, 2014.

Computer Aided Drafting, Design and Manufacturing Volume 26, Number 2, June 2016, Page 13

Computer Aided Drafting, Design and Manufacturing Volume 26, Number 2, June 2016, Page 13 Coputer Aided Drafting, Design and Manufacturing Volue 26, uber 2, June 2016, Page 13 CADDM 3D reconstruction of coplex curved objects fro line drawings Sun Yanling, Dong Lijun Institute of Mechanical