Color conversion method for multi-primary display for spectral color reproduction

Transcription

1 Journal of Electronic Imaging 13(4), (October 2004). Color conversion method for multi-primary display for spectral color reproduction Yuri Murakami Frontier Collaborative Research Center and Telecommunications Advancement Organization of Japan Akasaka Natural Vision Research Center Akasaka, Minato-ku Tokyo, , Japan Jun-ichiro Ishii Imaging Science & Engineering Laboratory Takashi Obi Interdiciplinary Graduate School of Science and Engineering Masahiro Yamaguchi Imaging Science & Engineering Laboratory and Telecommunications Advancement Organization of Japan Akasaka Natural Vision Research Center Akasaka, Minato-ku Tokyo, , Japan Nagaaki Ohyama Frontier Collaborative Research Center and Telecommunications Advancement Organization of Japan Akasaka Natural Vision Research Center Akasaka, Minato-ku Tokyo, , Japan Journal of Electronic Imaging / October 2004 / Vol. 13(4) / 701

2 Murakami et al. Abstract. In the conventional color reproduction based on the colorimetric match for a standard observer, color mismatch can be perceived if the color matching functions of the observer deviate from those of the standard observer; this phenomenon is known as observer metamerism. Recently, multi-primary display, using more than three-primary colors, has attracted attention as a color reproduction media because of its expanded gamut and its possibility to reduce the color mismatch caused by observer metamerism. In this paper, a new color conversion method for multi-primary display that reduces the observer metamerism is proposed. The proposed method gives the multi-dimensional control value of a display device to minimize the spectral approximation error under the constraints of tristimulus match. Reproduced spectrum by a seven-primary display is simulated and evaluated by the color matching functions of Stiles s 20 observers. The results confirmed that the proposed method reduces the color reproduction error caused by observer variability compared to the other seven-primary reproduction and conventional three-primary reproduction. The preliminary visual evaluation results with a seven-primary display using light-emitting diodes are also introduced SPIE and IS&T. [DOI: / ] 1 Introduction Along with the improvement and spread of various color imaging devices, we have more and more chances to see the displayed images of the objects that are placed at remote locations. The correspondence between the colors of the original object and the reproduced image is immensely important, especially in applications such as tele-medicine and on-line shopping. Recently, high fidelity color reproduction based on spectral information has been investigated. 1 3 In this method, the object is captured by a multispectral camera and the spectral reflectance of the object is estimated. However, spectral information has been utilized mainly to overcome the illumination metamerism problem; very few studies try to solve the observer metamerism by spectral information. Observer metamerism is a phenomenon known as follows. In the conventional color reproduction for the colorimetric match, the color matching functions of the CIE standard observer are used as a standard. That is, the reproduced light is controlled to have the same tristimulus values with the original object under a standard observer, but having the different spectrum. In this case, color mismatches may be perceived if the color matching functions of the observer deviate from those used in the color reproduction. Observer variability has been recognized since in the past, and the main result in the investigations is CIE standard-deviate observer, 4 which owes to the measurements by Stiles and Burch 5 in Recently, observer variability has been seriously reconsidered again, 6 10 the results of which show that inter-observer color mismatch was found to be significantly larger than previously thought. It is also reported that the color mismatch in crossmedia reproduction cannot be neglected for accurate color reproduction It is impossible to realize color matching for several deviated observers at the same time with conventional trichromatic displays, because the three-dimensional control value of the display device is determined uniquely for /2004/$ SPIE and IS&T. the reproduction of a set of tristimulus values. Recently multi-primary display was introduced and it attracted attention as a color reproduction media mainly due to its expanded gamut. 3,14 17 In the color reproduction by the multiprimary display, there are multiple choices of the control values for the reproduction of a set of tristimulus values. That is, this degree of freedom can be utilized to reduce the color mismatch caused by observer variability. For this purpose, color conversion methods have been proposed recently, 18,19 where color conversion implies the operation of obtaining a set of control values of a display device from device-independent color or spectrum information. Though those methods require a set of color matching functions that represents the population or individual, it is difficult to obtain the reliable data at the present day. In this paper, we propose a new color conversion method for multi-primary display to reduce the color mismatch caused by observer metamerism. The constraint used in the conversion is tristimulus match for the standard observer, which is the same constraint for the conventional color reproduction. Under this constraint, the proposed method does not need any information about the individual color matching functions or deviations to minimize the difference between the spectra of the original object and the reproduced light. The proposed method is applied to the simulation of the color reproduction using seven-primary color display and the effectiveness of the proposed method is evaluated using the color matching functions of Stiles s 20 observers. 5 We also present some results of the visual experiments, which ensure that the spectral-approximation reproduction reduces the color mismatch caused by the observer variability. 2 Method 2.1 Formulation of the Problem If the color generation of an N-primary display is based on the additive mixture of the primaries, the spectral intensity of the reproduced light P( ) is approximately represented by N P j 1 j p j, where p j ( ) (j 1,...,N) is the spectral intensity of the fullemitted jth primary light and j (0 j 1) is the control value of jth primary. In Eq. 1, it is assumed that the display satisfies both chromaticity constant and channel independence. Suppose S( ) is the spectral intensity reflected from an original object that we want to reproduce, the square error between S( ) and the reproduced spectrum by N-primary display is defined as E S P 2 d. 2 The aim of the proposed method is to determine a set of control values 1,..., N that minimizes E. When minimizing E, we impose the constraints that the tristimulus values of CIE standard observer are accurately reproduced, that is / Journal of Electronic Imaging / October 2004 / Vol. 13(4)

3 Color conversion method for multi-primary display... t k S d t k P d, k X,Y,Z, 3 where t k ( ) are color matching functions of CIE standard observer. This constraint is introduced because of the following reasons. If a set of control values is optimized only for spectral approximation, the tristimulus errors for most observers can be considerably large, especially when the number of the primaries is insufficient. To reduce the average mismatch, tristimulus match for the CIE standard observer is effective because CIE standard color matching functions are designed to represent the average color matching response of the population of human observers. Equation 2 is rewritten by Fig. 1 Schematic diagram of the seven-primary display using LED primaries. E S S d c T a 1 2 at Da, where N dimensional column vectors a and c, and N N matrix D are given by a 1,..., N T, c i 2 S p i d, D ij 2 p i p j d. Since the first term of Eq. 4 is constant, E can be replaced by E c T a 1 2 at Da. Equation 3 is also rewritten by g Fa, where three-dimensional column vector g and 3 N matrix F are given by g k t k S d, F kj t k p j d Finally, the problem results in a constrained nonlinear optimization which is composed of a quadratic objective function, linear equality and inequality constraints Objective function: Equality constraints: Inequality constraints: E cta 1 2 at Da, g Fa, 0 j Solution First, let us consider the optimization problem without the inequality constraints in Eqs. 12. This case can be solved by Lagrange multipliers; with a vector ( X, Y, Z ) T, we find the optimal point a by a D F 0 F T 1 c g, 13 where 0 is 3-by-3 zero matrix. This solution can be used only when a satisfies the inequality constraints in Eqs. 12. Next, let us consider the original optimization problem of Eqs. 12. There are several algorithms 20 to solve the nonlinear optimization problem with equality and inequality constraints, which require iterative operations, though. Typical algorithms are general reduced gradient GRG method and recursive quadratic programming method, by which we can obtain the solution of Eqs. 12. In these optimization algorithms, it is guaranteed that any local optimum is identical to the global optimum because both the objective function and the feasible region are convex. It is possible to reduce the calculation time for iteration to solve Eqs. 12, by the following procedures. First, a is calculated by Eq. 13, and it is adopted as the optimum only if it satisfies the inequality constraints. Otherwise, one of the abovementioned iterative methods should be applied to reach the real optimum. 3 Simulation and Experimental Results We developed the first prototype of seven-primary color display using light-emitting diodes LEDs, which generate colored light in the field 20 mm in diameter. The schematic diagram of the display is shown in Fig. 1. The display consists of 30 LEDs densely soldered on the circuit board, integrating sphere, LED controller and personal computer PC. The integrating sphere has two holes, and it is mounted on the LED-soldered circuit board so that the LEDs are inside the sphere through one of the two holes; another hole is used for the field of view. The light intensity of each LED can be modulated by the signals sent from the PC through the LED controller using pulse width modulation. The LED lights are mixed in the integrating sphere, and observed through the hole on the sphere. The spectra of the seven kinds of LEDs, P, B, BG, G, Y, YG, and R, are Journal of Electronic Imaging / October 2004 / Vol. 13(4) / 703

4 Murakami et al. Fig. 2 Spectra of the primaries of the seven-primary display using LED primaries. Fig. 3 XYZ color matching functions of CIE 1964 standard observer and one of the Stiles s 20 observers (#20). shown in Fig. 2. The spectra of the primaries are used in the simulations, and the seven-primary display is used in the visual experiment. 3.1 Simulation Results Color reproduction using the seven-primary display is simulated, in which the proposed color conversion method is applied. In the implementation of the proposed method, CIE 1964 color matching functions are used as the equality constraints and the GRG method is used as the nonlinear optimization algorithm. For comparison, we also simulate seven-primary reproduction with another color conversion method, matrix switching conversion method, 17 and threeprimary reproduction, which only uses B, G and R primaries shown in Fig. 2, with 3 3 color conversion matrix. In both color reproductions, the control values are calculated to match the CIE 1964 tristimulus values. As a set of object spectra, we use the spectral reflectances in the category of Paints in SOCS database, 21 since the samples in this category have various colors and spectral shapes. Illumination is assumed to be of CIE standard illuminant C, where the power is normalized so that the tristimulus values of the white object are included in the display gamut. The category of Paints contains 229 reflectances, where 221 samples are inside the gamut of the seven-primary display in the CIE 1964 color space under C illuminant. We use these 221 samples in the simulation; 56 of those samples are outside the gamut of the three-primary display where the control values are clipped to 0 or 1. The color reproduction error is measured using 20 of Stiles s 49 color matching functions 10 5,22 transformed to a CIE-type system corresponding closely to that in which the 1964 standard observer system is expressed. 22 Figure 3 shows color matching functions of CIE 1964 and Stiles s #20 observer; #20 corresponds to the observer OBJ The notation * (#k), k 1,...,20, is defined as the difference between the original and reproduced spectra in L*a*b* color space based on the color matching functions of #k observer. As a result of the proposed color conversion method, 49 of 221 samples are solved by Lagrange multipliers method. The rest of 172 samples require iterative GRG method to solve. We confirm that there is no color difference between the original and reproduced colors for CIE 1964 observer in 221 samples by both seven-primary reproductions, and in 165 samples by three-primary reproduction. Table 1 shows normalized root mean square error defined by NRMSE i S i P i 2 d j S j 2, 14 d where S i ( ) isith object spectrum and P i ( ) is the corresponding reproduced spectrum. From the result of Table 1, it is confirmed that the proposed method reduces the NRMSE of the reproduced spectra for the samples both inside and outside the three-primary gamut. We can see that with the proposed method the error of the samples outside Table 1 NRMSE of the reproduced spectra in the simulation. NRMSE of spectra Samples (number) Inside three-primary gamut (165) Outside three-primary gamut (56) a a Clipping of the control values is performed. 704 / Journal of Electronic Imaging / October 2004 / Vol. 13(4)

5 Color conversion method for multi-primary display... Fig. 4 Histogram of * (#20) of the samples inside three-primary gamut. Fig. 5 Layout of the setup for the visual experiment (up side view). the three-primary gamut is slightly larger than the samples inside. This is because we have less degree of freedom in the choice of the control values for the colors close to the surface of the gamut of the seven-primary display. For example, the control values of the color on the surface of the gamut are determined uniquely. Since the colors outside the three-primary gamut are highly saturated and close to the seven-primary gamut surface, the spectral approximation error becomes slightly large. Figure 4 shows the histogram of the color difference * (#20) of the 165 samples inside the three-primary gamut. While the color differences of most samples are included between 0 and 3 by the seven-primary reproduction of the proposed method, non-negligible samples spread in the region more than three by the matrix switching and three-primary reproduction. This result shows that the reduction of the error in the spectral approximation reduces the error in the color space for the deviated observer. Table 2 shows the average and maximum * (#k) over both 20 observers and 165/56 samples. First, it should be noted that the error in the three-primary reproduction for the samples outside the gamut is considerably large because the control values are clipped and the tristimulus values for the standard observer do not match to the original. Among the three reproductions, we can see that the seven-primary reproduction by the proposed method reduces the average and maximum * (#k) to about 1/2 to those of matrix switching method and to about 1/5 to those of threeprimary. 3.2 Visual Evaluation In this section, the effectiveness of the proposed method is evaluated through visual experiments. Illuminated color patches are reproduced on the seven-primary display with three kinds of color reproduction method used in the simulations, and the reproduction which is visually close to the original patch is examined through visual experiments. The experimental setup is shown in Fig. 5. The distance between the observer and the screen is about 40 cm. The screen is white and the room light is on, so that the observers are light adapted. The horizontally adjacent two 0.8-cmsquare windows are on the screen, each one of which subtends a visual angle of 1.1. With the use of a half mirror, the observers are able to view the color patch and the reproduced color through the adjacent windows. The color patch is seen through the right window and the reproduced light is seen through the left window. A color patch is placed in the light box and illuminated by the illumination with a color temperature of 6500 K. In this experiment, bluish green and magenta in Macbeth ColorChecker are used. Before the visual evaluation, the following preparations are done. The spectra of the color patches are measured by a Photo Research PR-704 spectroradiometer, where PR-704 is placed at the same place as the observers eye. A set of Table 2 Simulation results. Average and maximum * (#k) of the reproduced spectra over the object reflectances and Stiles s 20 observers. Average * (#k) Maximum * (a) (k) Samples (number) Inside three-primary gamut (165) Outside three-primary gamut (56) a a a Clipping of the control values is performed. Journal of Electronic Imaging / October 2004 / Vol. 13(4) / 705

6 Murakami et al. Table 4 Visual experimental results. (a) is seven-primary reproduction with the proposed method, (b) is seven-primary reproduction with the matrix switching method, (c) is three-primary reproduction. Each figure indicates the number of the observers who select the respective reproduction that is perceived to be close to the original. Number Bluish green Magenta (a)-(b) (a) (b) Total (a) (b) Total (a)-(c) (a) (c) Total (a) (c) Total Fig. 6 The spectral distributions of the bluish green patch and the reproduced results. control values of the seven-primary display is calculated for two color patches by a proposed method, b matrixswitching method, and c 3 3 matrix conversion supposing only B, G and R primaries are used. Every method is undertaken to realize the tristimulus match based on the CIE1931 color matching functions. The reproduced color is measured using PR-704 in the same way as the color patches are measured. The spectral distributions of the bluish green patch and the reproduced results are shown in Fig. 6. The NRMSE of spectra and * using CIE 1931 color matching functions, * (1931), are summarized in Table 3. It can be seen that for bluish green the NRMSE of the proposed method is less than that of the matrix-switching method. On the other hand, both the seven-primary reproductions have almost the same NRMSE for magenta. For both color patches, the NRMSE of three-primary reproduction is larger than those of seven-primary reproductions. Despite the color conversions aiming at the tristimulus match, there is still error in * (1931), which is mainly because of the characterization error of seven-primary display based on Eq. 1. However, this error is not much of a problem in this visual evaluation because far larger color differences are supposed to be perceived among different methods by actual observers when the spectral differences are large as shown in Fig. 6. Six observers for bluish green and four observers for magenta participated in the visual experiment. In one session, the observer is asked to select a stimulus closer to the stimulus in the right window from the two stimuli appearing in the left window alternatively. The observer is able to switch the two stimuli in the left window with a keyboard, but does not know which stimulus in the left window corresponds to which reproduction method. The comparisons a - b and a - c for two color patches are done; four sessions in all per observer. The results are shown in Table 4. In the results of bluish green, all six observers except one selected the proposed method as a reproduction that is closer to the original in both comparison a - b and a - c. In Magenta, all observers selected the proposed method in the comparison a - c, but the same number of observers selected a and b, respectively, in the comparison a - b. From those results, it is confirmed that the small-nrmse reproduction is visually close to the original. 4 Discussions In the simulations and experiments described in this paper, the primary colors of LEDs are used, but the relation between the spectral shapes of a set of primaries and the degree of the color mismatch was not discussed. In the results in Table 2, the maximum error in the seven-primary reproduction with the proposed method is nearly 6.0. Considering that the variability of the Stiles s data is supposed to be smaller than the actual, this error is considerably large. The reason for this error is thought to be as follows; LED primaries have little spectral power at around 550 and 620 nm as seen in Fig. 2, and they are clearly not appropriate to represent the spectra of paints. In this manner, the inappropriate primaries to represent the spectra of real objects could bring large disagreement between the reproduced image and the original object even when the proposed method is applied. Since most natural objects have Table 3 Experimental results. NRMSE and * (1931) of the reproduced spectra of bluish green and magenta color patches. NRMSE of spectra * (1931) Color patch Bluish green Magenta / Journal of Electronic Imaging / October 2004 / Vol. 13(4)

7 Color conversion method for multi-primary display... smooth spectrum shape, the large error is supposed to more likely arise when the bandwidths of primary spectra are narrow and when primary spectra has spikes, such as laser and fluorescent light. From this viewpoint, the design of the spectra for the primaries will be an important research issue in the future. Next, we mention that the improvement of the proposed method is possible by utilizing the information on the variability of the color matching functions. Although the proposed method can be applied without using the color matching function data representing the observer variability, the proposed method can also be extended if we can obtain the reliable data of the variability information. For example, if we have the K sets of the color matching functions and want to optimize with respect to these color matching functions, instead of Eq. 2, it is appropriate to determine the objective function by the mean square error of the stimuli for K sets of color matching functions. That is E w 1 3K S w k P w k 2, 3K k 1 where S w k w k S d, P w k w k P d and w k ( ), k 1,...,3K, is the one of the color matching functions of an observer. In this case, the color conversion method also becomes the optimization problem written by Eqs. 12, but c and D are redefined as c i 2 S R, p i d d, D ij 2 p i R, p j d d, where R, 1 3K w 3K k w k. k Therefore, optimization based on Eq. 15 can be solved in the same framework presented in this paper. 5 Conclusions This paper proposed a color conversion method for multiprimary display to minimize the spectral approximation error under the constraints of tristimulus match. We presented that the color conversion method for spectral approximation can be formulated by the convex nonlinear optimization problem. Lagrange multipliers solution is also introduced for the reduction of the computation time for iteration. Through the simulation of the color reproduction using seven-led-primary display, it is confirmed that the proposed method reduces the NRMSE of the reproduced spectra and that the reduction of the spectral approximation error is effective in reducing the color mismatch caused by observer variability; average and maximum * (#k) over object reflectances and observers are reduced to about 1/2 and 1/5 to those of matrix switching method and threeprimary reproduction, respectively. Since the set of LEDs used in the simulation is not exactly appropriate for this purpose, another primaries whose spectra cover the range of the visible wavelength could further improve the results of the proposed method. Through the preliminary visual experiments, we confirm that the spectral-approximation reproduction is effective for visual matching if the color matching functions of the observer deviate from the standard; similar results are also reported in the visual experiments using a six-primary projection display. 13 References 1. Th. Keusen, Multispectral color system with an encoding format compatible with the conventional tristimulus model, J. Imaging Sci. Technol. 40, H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura, and Y. Miyake, System design for accurately estimating spectral reflectance of art paintings, Appl. Opt. 39, M. Yamaguchi, T. Teraji, K. Osawa, T. Uchiyama, H. Motomura, Y. Murakami, and N. Ohyama, Color image reproduction based on the multispectral and multiprimary imaging: Experimental evaluation, Proc. SPIE 4663, CIE Publ. No. 80, Special Metamerism Index: Change in Observer, Central Bureau of the CIE, Vienna W. S. Stiles and J. M. Burch, N.P.L. colour-matching investigation: Final report, Opt. Acta 6, W. A. Thornton, Toward a more accurate and extensible colorimetry. Part I. Introduction. The visual colorimeter-spectroradiometer. Experimental results, Color Res. Appl. 17 2, W. A. Thornton, Toward a more accurate and extensible colorimetry. Part II. Discussion, Color Res. Appl. 17 3, W. A. Thornton, Toward a more accurate and extensible colorimetry. Part III. Discussion continued, Color Res. Appl. 17 4, A. D. North and M. D. Fairchild, Measuring color-matching functions. Part I, Color Res. Appl. 18 3, A. D. North and M. D. Fairchild, Measuring color-matching functions. Part II. New data for assessing observer metamerism, Color Res. Appl. 18 3, D. C. Rich and J. Jalijali, Effects of observer metamerism in the determination of human color-matching functions, Color Res. Appl. 20 1, R. L. Alfvin and M. D. Fairchild, Observer variability in metameric color matches using color reproduction media, Color Res. Appl. 22 3, K. Ohsawa, T. Teraji, F. Koenig, M. Yamaguchi, and N. Ohyama, Color matching experiment using 6-primary display, in Proc. 3rd Int. Conf. on Multispectral Color Science T. Ajito, T. Obi, M. Yamaguchi, and N. Ohyama, Multiprimary color display for liquid crystal display projectors using diffraction grating, Opt. Eng , T. Ajito, T. Obi, M. Yamaguchi, and N. Ohyama, Expanded color gamut reproduced by six-primary projection display, Proc. SPIE 3954, B. Hill, Optimization of total multispectral imaging systems: best spectral match versus least observer metamerism, in Proc. SPIE 4421, T. Ajito, K. Ohsawa, T. Obi, M. Yamaguchi, and N. Ohyama, Color conversion method for multiprimary display using matrix switching, Opt. Rev. 8 3, K. Ohsawa, F. König, M. Yamaguchi, and N. Ohyama, Multiprimary display optimized for CIE1931 and CIE1964 color matching functions, Proc. SPIE 4421, F. König, K. Ohsawa, M. Yamaguchi, and N. Ohyama, A multiprimary display: Discounting observer metamerism, Proc. SPIE 4421, D. M. Simmons, Nonlinear Programming for Operations Research, Prentice Hall, Englewood Cliffs, NJ TRX0012, JIS G. Wyzcecki and W. S. Stiles, Color Science: Concepts, Methods, Quantitative Data and Formulas, Wiley, New York Journal of Electronic Imaging / October 2004 / Vol. 13(4) / 707

8 Murakami et al. Yuri Ohya Murakami received her MS degree in information processing from (Japan) in Since 2000 she has been a research associate in the Frontier Collaborative Research Center of the and a fellow researcher of Akasaka Natural Vision Research Center, TAO (Japan). She is working in the fields of color image reproduction using multispectral and multi-primary imaging and multispectral image compression. Jyun-ichiro Ishii received his MS degree in information processing from in After graduation, he joined the NEC Corp. Takashi Obi is an associate professor at the Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology. He received his PhD in information processing from in He is working in the fields of medical imaging and information security systems. Masahiro Yamaguchi is an associate professor at the Imaging Science & Engineering Laboratory of the and a sub leader of the Akasaka Natural Vision Research Center. He received his PhD in information processing from the Tokyo Institute of Technology in He is working in the fields of 3D imaging, color imaging, optics and information security systems. Nagaaki Ohyama is a professor at the Frontier Collaborative Research Center of the and a leader of the Akasaka Natural Vision Research Center. He received his PhD in information processing from the in He is working in the fields of optics, medical imaging and information security systems. 708 / Journal of Electronic Imaging / October 2004 / Vol. 13(4)