FCV216, the 22nd Korea-Japan Joint Workshop on Frontiers of Computer Vision Chromatiity-mathed Superimposition of Foreground Objets in Different Environments Yohei Ogura Graduate Shool of Siene and Tehnology Keio University y.ogura@hvrl.is.keio.a.jp Yoihi Nakasone Huawei Tehnologies Japan Nakasone.Yoihi@huawei.om Shao Ming Huawei Tehnologies Japan Shao.Ming@huawei.om Hideo Saito Graduate Shool of Siene and Tehnology Keio University saito@hvrl.is.keio.a.jp Abstrat The illumination hromatiity is one of the important fator for appearane of objets in images. When we superimpose some objets on another sene to generate omposite images, the resulting image may give us strangeness if the illumination hromatiity of foreground sene and bakground sene are different. In this paper, we propose a omposite image synthesis method whih an ahieve illumination hromatiity onsisteny. Our goal is to generate a illumination hromatiity-mathed omposite image from two images whih are taken under different illumination environment. We estimate hromatiity of two senes based on segmented olor regions. Subsequently illumination hromatiity mathing is performed based on estimated hromatiity of two senes. After the hromatiity mathing proess, we extrat the target person by using the grab ut segmentation. The result image is obtained by superimposing the target person or objet onto bakground sene. We demonstrate that our method an perform some different illumination hromatiity environment. I. INTRODUCTION Making a omposite image is an important funtionality of image editing for entertainment experienes. Typial appliation is superimposing a person into a different image that aptures different sene. However, we may feel something is strange if the illumination olor of two senes are different. Illumination hromatiity is one of the most essential fator that deides how a person in a image looks like. Therefore, we should ahieve the onsisteny of illumination hromatiity of a foreground objet and a bakground sene when we make omposite images. Some illumination hromatiity estimation methods based on the assumption of Dihromati Refletion Model as refletane property of the target sene. have been proposed. The dihromati refletion model represents surfae refletane as aombination of an objet olor by diffuse refletion and a illumination olor by speular refletion. We an assume that a most of everyday things, suh as things made of plasti, usually have speular refletion, but there are some exeption suh as general lothing with low speular refletion. The dihromati refletion model is widely used as the model whih an approximate surfae refletane of general things. This model was proposed by Shafer et al. [1]. Klinker et al. showed that it is possible to apply the dihromati refletion model to real senes and to determine the illumination olor from the model [2]. Lehmann et al. proposed a method that applied the two-dimensional dihromati refletion model to the regions around the speular highlights in a olor image, and estimated the illumination hromatiity [3]. Finlayson et al proposed illumination hromatiity estimation method using the blak-body radiator in two-dimensional spae. [4]. Tajima showed the illumination hromatiity estimation using the two dimentional dihromati refletion model with imperfet image segmentation [6]. Tajima also showed that the two dimentional dihromati refletion model an be applied to apparently lambertian surfae with multiple olors. In this paper, we propose a omposite image synthesis method whih an ahieve illumination hromatiity onsisteny. Our goal is to generate a illumination hromatiitymathed omposite image from two images whih are taken under different environment. We estimate illumination hromatiity of two senes from multiply segmented region whih have different olors. The illumination hromatiity estimation is based on the method proposed by Tajima [6]. Subsequently illumination hromatiity mathing is performed based on estimated hromatiity of two senes. After the hromatiity mathing proess, we extrat the target person or objet region by using the grab ut segmentation algorithm. Finally, the result image is obtained by superimposing the target person or objet onto bakground sene. II. PROPOSED METHOD Our goal is to ahieve illumination hromatiity onsisteny on omposite images from two images whih are taken under different environment. At first, illumination hromatiity of input images are estimated from multiple olor segmented region. This proess is based on the work proposed by Tajima [6]. We estimate illumination hromatiity from a olor image onsists from multiple olor. The input olor images are segmented into multiple region and estimate illumination hromatiity on eah input image. After the illumination hromatiity estimation proess, we adjust the hromatiity of the sene whih has a target person or objet to be superimposed. Subsequently, the region of target person or objet is subtrated based on the grab ut segmentation. Finally, the target person or objet is superimposed on bakground sene. -141-
A. Illumination Chromatiity Estimation same region beause the illumination hromatiity estimation is based on this assumption. After the olor segmentation, we estimate illumination hromatiity. We assume that illumination hromatiity is onstant throughout the input image and illumination intensity is not taken into aount. First, we onvert the input olor image from RGB olor spae to CIE-XYZ olor spae to obtain eah value (X, Y, Z). Subsequently we alulate the hromatiity by using the following equation. X Y, y= (1) X +Y +Z X +Y +Z Sine we have obtained hromatiity values on eah point, we plot the lustered points (x, y) whih are assigned in eah segmented region on the CIE-xy hromatiity diagram. The prinipal omponent analysis (PCA) algorithm is applied to eah lusters and obtain the first prinipal omponent and the entroid. Illumination hromatiity is assumed to lie on the line whih orresponds the diretion of first prinipal omponent and the entroid of the luster. The hromatiity lines an be expressed as shown in Equation (2). x= aj y + bj x = j Fig. 1. Target region extration and superimposition Before the explanation of illumination hromatiity estimation, we define the name of the input sene. Sr sene is the sene whih has the objet to be superimposed. Dst sene is the sene to be the bakground of the omposite image. Our hromatiity estimation part is based on a method by Tajima [6]. At first, the input olor images are onverted from RGB olor spae into HSV olor spae. The olor segmentation is done in five dimensional spae: hue, saturation and value from HSV olor spae and x, y from olor image oordinate. Eah of these five properties has different value range so eah property is weighted to get appropriate segmentation result. The important point is that objets having the same olor and onsisting from the same material should be divided into the Fig. 2. Input image for olor segmentation the result of olor segmentation (2) aj bj j are the oeffiients of the jth hromatiity line. So we an estimate the illumination hromatiity from an intersetion of all lines. However, all lines do not always interset eah other at a single point. Therefore, we estimate the point (xl, yl ) that minimizes the distane between all lines and point (xl, yl ) by using the least square method. We an express Equation (2) in a matrix system as shown in Equation (3) 1 a 1 b1 2 a 2 b2 x, x =, =. (3) A=.... y.... an bn n We minimize the error Ax 2 and obtain (xl, yl ). After that, (xl, yl ) is onstrained to the urve of the blakbody radiator. Natural light soures suh as sunlight or light from light bulbs an be approximated by the blak-body radiator. The power spetrum E(λ, T ) of the blak-body radiator an be shown by Equation (4) E(λ, T ) = 2h2 1 h λ5 e kt λ 1 (4) T is the temperature of the blak-body in Kelvin. h is the plank onstant and k is the Boltzmann onstant. shows the speed of light. We projet the urve of the blak-body radiator(equation (4)) onto CIE-xy hromatiity diagram and obtain the quadrati funtion [7]. Sine we have a point (xl, yl ) whih minimizes the distane between all lines, the final result of illumination hromatiity (x, y ) an be obtained by estimating the nearest point on the blak-body radiator urve from (xl, yl ). This illumination hromatiity estimation proess is done on both the foreground sene and the bakground sene. -142-
B. Chromatiity Adjustment We have estimated the illumination hromatiity as desribed in the setion II-A. The purpose of this setion is to ahieve the hromatiity onsisteny between the sr sene image and the dst sene image. The illumination hromatiity (x, y ) that we have estimated are normalized values. z an be obtained easily from (x, y ) and Equation (5) x + y + z = 1 (5) sr sr Here we an get hromatiity of sr sene (xsr, y, z ) dst dst dst and dst sene (x, y, z ). Then, the hromatiity of sr sene is adjusted to math the hromatiity of dst sene by the ratio of the hromatiity shown in Equation (6). xdst Ix (p) xsr Ixout (p) in ydst Iyin (p) (6) Iyout (p) = sr y Izout (p) Izin (p) z dst zsr (Ixin (p), Iyin (p), Izin (p)) indiate input pixel values whih onverted to CIE-XYZ olor spae at a point p and (Ixout (p), Iyout (p), Izout (p)) shows output pixel values respetively. This proessing is applied to all pixels of the input image. sr sr dst dst dst Note that (xsr, y, z ) and (x, y, z ) are normalized values. This means that it shows the ratio of (X, Y, Z) values. Therefore, we don t take into aount the differene of the illumination intensity(brightness) on sr sene and dst sene. The input image and adjusted image are shown in Figure II-B. Fig. 4. Target region extration and superimposition () Fig. 3. Sr sene before hromatiity adjustment. After hromatiity adjustment. Fig. 5. Input image After the first iteration of grab ut segmentation () user input of touh up proess final segmentation result C. Target Region Extration and Superimposition tagged image and the final extrated result image are shown in Figure 5. Our goal is to make omposite images so the boundary of the target objet and bakground sene is very important. We apply the alpha blending around the boundary of the objet when the target objet is superimposed on dst sene. We onsider the illumination hromatiity onsisnteny only and don t take are of geometrial onsisteny in our method. We have obtained adjusted olor image of sr sene. In this setion, we introdue the target region extration and superimposition on dst sene. In our method, we use the grab ut segmentation algorithm [8] to extrat the target region. Firstly, a target region is seleted with a retangular and the first iteration is performed subsequently. If there are some redundant regions or regions whih are needed but removed in the first iteration result, users an touh up the segmented result. Users an set four tags: definitely foreground, definitely bakground, probably foreground and probably bakground. After the tagging, the seond iteration is performed. This iterative proess is performed until the extration result is appropriate. An input image, the first iteration result, the III. E XPERIMENTS In this setion, we present experiments performed to demonstrate the effetiveness of the proposed method. We apture an image of a person standing in arbitrary illumination environment. Illumination hromatiity of the sene is unknown in -143-
this ase. We estimate illumination hromatiity of Sr sene and Dst sene, and adjust the Sr sene s hromatiity to fit to Dst sene. Finally, we get the final result by superposing the person to the bakground images. A. Experiment ondition We aptured a person under slightly amber illumination for sr sene. Dst sene is the other plae and onsists from slightly bluish illumination for Pattern 1, as shown in Figure 6. For Pattern 2, sr sene is aptured in indoor sene whih has fluoresent lamps. Dst sene is the same sene as sr sene of Pattern 1, whih has slightly amber illumination as shown in Figure 7. Fig. 6. Input images for experiment 1. Sr sene Dst sene The result for Pattern 2 are shown in Figure 9. Illumination hromatiity of Sr sene is estimated as white and that of Dst sene is estimated as a little amber, as shown in Figure 9, (e) and (f). Comparing Figure 9, and (), our method ould estimate and math the hromatiity and generate a omposite image with illumination hromatiity onsisteny. IV. CONCLUSION In this paper, we proposed a omposite image synthesis method whih an ahieve illumination hromatiity onsisteny. Illumination hromatiity of two senes were estimated from multiple segmented regions with different olors. Using multiple olor regions, the illumination hromatiity an be estimated even without rih speular refletion. Subsequently illumination hromatiity mathing was performed based on estimated hromatiity of two senes. After the hromatiity mathing proess, we segment the target person or objet on the grab ut segmentation algorithm. Finally, the result image is obtained by superimposing the target person or objet onto bakground sene. In experiment, we tested our method to show the utility and quantitative evaluation. The future work are the quantitative evaluation and making our method more robust and applying our method to light soures whih drastially stay off the blak body radiator. REFERENCES Fig. 7. Input images for experiment 2. Sr sene Dst sene Illumination hromatiity is estimated by using eah segmented region inluding not only human region, but also bakground area. The number of segmented region for hromatiity estimation is 48, and regions whose size is less than 1 pixels is exluded in the illumination hromatiity estimation. Sine we don t onsider the illumination intensity mathing, the brightness of sr sene is adjusted manually. B. Experiment result The result images of Pattern 1 are shown in Figure 8. Illumination hromatiity for a man standing on the left side and that of bakground sene is different in result without hromatiity adjustment (Figure 8) omparing to the ground truth whih shows the target person atually standing in dst sene. Applying our method, the final result ahieved illumination hromatiity onsisteny between a superimposed and bakground sene. Fousing on a pants of the superimposed person, we an say that the hromatiity mathing proess is done orretly. As shown in Figure 8 and (e), eah line is obtained from pixel values from eah segmented area. They show us that we ould estimate the hromatiity from multiple region whih labeled as different olor region. [1] S. A. Shafer, Using olor to separate refletion omponents. Color Researh & Appliation, vol. 1 no. 4, pp.21-218, 1985. [2] G. J. Klinker, S. A. Shafer, and T. Kanade, The measurement of highlights in olor images. International Journal of Computer Vision, vol. 2 no.1, pp.7-32, 1988. [3] T. M. Lehmann and C. Palm, Color line searh for illuminant estimation in real-world senes. Journal of the Optial Soiety of Ameria A, vol. 18 no.11, pp.2679-2691, 21. [4] G. D. Finlayson and G. Shaefer, Solving for olour onstany using a onstrained dihromati refletion model. International Journal of Computer Vision, vol. 43 no. 3, pp.127-144, 21. [5] M. Ebner and C. Herrmann, On determining the olor of the illuminant using the dihromati refletion model. Pattern Reognition, pp.1 8, 25. [6] J. Tajima, Illumination hromatiity estimation based on dihromati refletion model and imperfet segmentation. Computational Color Imaging, pp.51-61, 29. [7] B. Kang and O. Moon and C. Hong and H. Lee and B. Cho and Y. Kim, Design of advaned olor - Temperature ontrol system for HDTV appliations. Journal of the Korean Physial Soiety, 41.6, pp.865-871, 22. [8] C. Rother and V. Kolmogorov, and A. Blake. Grabut: Interative foreground extration using iterated graph uts. ACM Transations on Graphis (TOG), vol. 23 no. 3, pp.39-314, 24. -144-
() (e) (f) Fig. 8. Results for experiment 1. Result image Result without hromatiity adjustment () Ground truth Estimated hromatiity on Sr sene (e) Estimated hromatiity on Dst sene (f) Chromatiity mathing, a irle for sr hromatiity and an asterisk for dst hromatiity (e) () (f) Fig. 9. Results for experiment 1. Result image Result without hromatiity adjustment () Ground truth Estimated hromatiity on Sr sene (e) Estimated hromatiity on Dst sene (f) Chromatiity mathing, a irle for sr hromatiity and an asterisk for dst hromatiity -145-