Foveated Wavelet Image Quality Index *

Transcription

1 Foveated Wavelet Image Quality Index * Zhou Wang a, Alan C. Bovik a, and Ligang Lu b a Laboratory or Image and Video Engineering (LIVE), Dept. o Electrical and Computer Engineering The University o Texas at Austin, Austin, TX , USA b Video and Image Systems, IBM T. J. Watson Research Center, Yorktown Heights, NY 0598, USA zwang@ece.utexas.edu, bovik@ece.utexas.edu, lul@us.ibm.com ABSTRACT The human visual system (HVS) is highly non-uniorm in sampling, coding, processing and understanding. The spatial resolution o the HVS is highest around the point o ixation (oveation point) and decreases rapidly with increasing eccentricity. Currently, most image quality measurement methods are designed or uniorm resolution images. These methods do not correlate well with the perceived oveated image quality. Wavelet analysis delivers a convenient way to simultaneously examine localized spatial as well as requency inormation. We developed a new image quality metric called oveated wavelet image quality index (FWQI) in the wavelet transorm domain. FWQI considers multiple actors o the HVS, including the space variance o the contrast sensitivity unction, the spatial variance o the local visual cuto requency, the variance o human visual sensitivity in dierent wavelet subbands, and the inluence o the viewing distance on the display resolution and the HVS eatures. FWQI can be employed or oveated region o interest (ROI) image coding and quality enhancement. We show its eectiveness by using it as a guide or optimal bit assignment o an embedded oveated image coding system. The coding system demonstrates very good coding perormance and scalability in terms o oveated objective as well as subjective quality measurement. Keywords: image quality assessment, human visual system (HVS), oveation, wavelet, image coding. INTRODUCTION The photoreceptors (cones and rods) and ganglion cells are non-uniormly distributed in the retina in the human eye []. The density o cone receptors and ganglion cells plays important role in determining the ability o our eyes to resolve what we see. Spatially, the visual resolution is highest around the point o ixation (oveation point) and decreases rapidly as a unction o eccentricity. Consequently, the human visual system (HVS) is highly spatial-variant in sampling, coding, processing and understanding visual inormation. The motivation behind oveation image processing is that there exists considerable high requency inormation redundancy in the peripheral regions, thus a much more eicient representation o images can be obtained by removing or reducing such inormation redundancy, provided the oveation point(s) and the viewing distance can be discovered. There has been growing recent interest in research work on oveated image processing [-5], including oveation iltering and oveated image and video compression. Currently, most image quality measurement methods are designed or uniorm resolution images. However, little has been done in the assessment o non-uniorm resolution images. For example, peak signal-to-noise ratio is still used or region o interest (ROI) image coding and postprocessing [6, 7]. Quality assessment method is very important or oveated image coding, because image coding is essentially an optimization procedure that attempts to maximize image quality with a limited number o bits, where the quality metric servers as a guide or bit assignment. Quality metrics are * This research was partially supported by IBM Corporation, Texas Instrument, Inc., and Texas Advanced Technology Program.

2 also very useul or the design o quality enhancement algorithms or the preprocessing and postprocessing o images. Wavelet analysis delivers a convenient way to simultaneously examine localized spatial as well as requency inormation. In this paper, we present a oveation-based HVS image quality metric, namely oveated wavelet image quality index (FWQI), in the discrete wavelet transorm (DWT) domain.. FOVEATED VISUAL SENSITIVITY MODEL Let us irst examine the anatomy o the early vision system. The light irst passes through the optics o the eye and is then sampled by the photoreceptors on the retina. There are two kinds o photoreceptors cones and rods. The cone receptors are responsible or daylight vision. Their distribution is highly non-uniorm on the retina. The density o the cone cells is highest at the ovea and drops very ast with increasing eccentricity. The photoreceptors deliver data to the bipolar cells, which in turn supply inormation to the ganglion cells. The distribution o ganglion cells is also highly nonuniorm. The density o the ganglion cells drops even aster than the density o the cone receptors. These density distributions play important roles in determining the resolution ability o the human eye. Psychological experiments had been conducted to measure the contrast sensitivity as a unction o retinal eccentricity [, 8-9]. In [], a model that its the experimental data was given as e + e CT (, e) = CT0 exp α, () e where is the spatial requency (cycles/degree), e is the retinal eccentricity (degrees), CT 0 is a constant minimal contrast threshold, α is the spatial requency decay constant, e is the hal-resolution eccentricity, and CT(, e) is the visible contrast threshold as a unction o and e. The best itting parameter values given in [] are α = 0.06, e =.3, and CT 0 = /64. It was also reported in [] that the same values o a and e provide a good it to the data in [8] with CT 0 = /75, and an adequate it to the data in [9] with CT 0 = /76, respectively. We use the parameter selections as in []. The contrast sensitivity is deined as: CS (, e) =. () CT (, e) For a given e, equation () can be used to ind its critical requency or so called cut-o requency c by setting CT to.0 (the maximum possible contrast) and solving or e e ln( CT0 ) c = (cycles/degree). (3) ( e + e )α Assume that the observed image is N pixels wide and the line rom the ovea to the point o ixation in the image is T perpendicular to the image plane. Also assume that the position o the oveation point x = x, x ) (pixels) and the ( viewing distance v (measured in image width) rom the eye to the image plane are known. The distance u (measured in image width) rom point T x = ( x, x ) to x is then u = d(/n, where d( = x x = ( x x ) + ( x x ) (measured in pixels). The eccentricity is given by u d( e( v, x ) = tan = tan. (4) v Nv Fig. shows the normalized contrast sensitivity as a unction o pixel position or N = 5 and v =, 3, 6 and 0, respectively. The cut-o requency as a unction o pixel position is also given. The contrast sensitivity is normalized so that the highest value is always.0 at 0 eccentricity. It can be observed that the cut-o requency drops quickly with increasing eccentricity and the contrast sensitivity decreases even aster. In real world digital images, the maximum perceived resolution is also limited by the display resolution r:

3 Spatial Frequency (cycles/degree) Spatial Frequency (cycles/degree) Pixel Position (pixels) Pixel Position (pixels) Spatial Frequency (cycles/degree) Spatial Frequency (cycles/degree) Pixel Position (pixels) Pixel Position (pixels) Fig. Normalized contrast sensitivity (Brightness indicates the strength o contrast sensitivity). The top-let, top-right, bottom-let and bottom-right igures are or N = 5 and viewing distance v =, 3, 6 and 0 times o the image width, respectively. The white curves show the cuto requency. πnv πe πnv N v πnv r = sec = (pixels/degree) (5) d ( + N v 80 This approximation is equivalent to that given in [0]. According to the sampling theorem, the highest requency that can be represented without aliasing by the display, or the display Nyquist requency, is hal o r: r πnv d = (cycles/degree). (6) 360 Combining (3) and (6), we obtain the cuto requency or a given location x by: m ( x, v) = min( ( e( x, v)), ( v)) c d 39.3 min tan = πnv, ( d( Nv) 360. (7) At a small viewing distance such as v =, the display Nyquist requency is so small that the cuto requency stays almost unchanged or a large range o eccentricities. However, strong oveation is still obtained because the contrast sensitivity is very sensitive to eccentricity, as shown in Fig.. Finally, we deine the oveation-based error sensitivity or

4 given viewing distance v, requency and location x as: CS(, e( v, ) = exp( e( v, ) or m ( S ( v,, = CS(,0). (8) 0 or > ( S is normalized so that the highest value is always.0 at 0 eccentricity. m 3. FOVEATED IMAGE QUALITY METRIC The DWT has proved to be a powerul tool or image processing and coding. In the -D DWT, the input discrete signal s is convolved with highpass and lowpass analysis ilters and downsampled by two, resulting in transormed signals s H and s L. The signal s L can be urther decomposed and the process may be repeated multiple times. The number o repetitions deines the wavelet decomposition level λ. For image processing, the horizontal and vertical wavelet decompositions are applied alternatively, yielding LL, HL, LH and HH subbands. The LL subband may be urther decomposed and the process repeated multiple times. A typical DWT decomposition structure is given by Fig.. Let ( λ, ) represent the subband o level λ and orientation, where is an index representing the LL, LH, HH or HL subband. The wavelet coeicients at dierent subbands supply inormation o variable perceptual importance. In [0], psychovisual measurement results were given or the visual sensitivity in wavelet decompositions. A model that its the experimental data is [0] log Y = log a + k(log log g ), (9) 0 LL3 HL3 LH3 HH3 LH LH HL HH HL HH Fig. DWT decomposition structure. where Y is the visually detectable noise threshold and λ = r [0] is the spatial requency. For gray scale models, a is [0] 0.495, k is 0.466, 0 is 0.40, and g is.50,, and or the LL, LH/HL, and HH subbands, respectively. The error sensitivity in subband ( λ, ) is given by: where S W A ( λ, ) = A λ, is the basis unction amplitude given in [0]. Let λ, A λ,, = λ, (0) λ Y k (log( 0 g / r ) λ, Aλ, a0 residing in subband ( λ, ). For each subband, we calculate the corresponding oveation point LL: T x x λ, =, λ λ x ; LH: B denote the set o wavelet coeicient positions T x + N x x λ =, ;, λ λ x λ, in it:

5 Given a wavelet coeicient at by d λ λ, ( λ, + HL: x x N x λ =, ; HH:, λ λ x B λ, = x x. With this distance, we have T T + x + N x N λ, =, λ λ x. (), its equivalent distance rom the oveation point in the spatial domain is given S (,, (, λ v = S v r, dλ, ( ) or x B λ,. () A oveation-based error sensitivity model in the DWT domain is obtained by combining (0) and (): β [ ] [ ] λ (, ) (,, ( )) β S λ S v r d x where β and β are parameters used to control the magnitudes o S( v, = or w λ, x B λ,, (3) S w and S, respectively. We use β = and β =.5. Fig. 3 shows S ( v, or v =, 3, 6 and 0, respectively. We deine a oveated wavelet image distortion (FWD) metric as: FWD = M M Q [ S( v, ) c( ) c ( ) ] n x n xn n= Q x, (4) where M is the number o the wavelet coeicients, and c x ) and c x ) are the n-th wavelet coeicients o the original and the compressed images at location x in the DWT domain, respectively. Q is set to in our measurement n system. We deine a oveated wavelet image quality index (FWQI) as: FWQI = exp( FWD). (5) The value o FWQI is between 0 and, with the maximum value at FWD = 0. Because S v, x ) varies with the viewing distance v, both FWD and FWQI o a test image are unctions o v, instead o single values. ( n ( n ( n Fig. 3 Foveation-based error sensitivity mask in the DWT domain. The top-let, topright, bottom-let, and bottom-right igures are or viewing distance v =, 3, 6 and 0 times o the image width, respectively. (Brightness logarithmically enhanced or display purpose) Although there is only one oveation point at one time or one human observer, it is necessary to allow multiple

6 oveation points in practice to provide more lexibility and robustness. This is because ) the usual pattern o human ixation is that the ixation point moves slightly around a small area o the center point o interest, ) there may be multiple human observers watching the image at the same time, and 3) there may exist multiple points and/or regions in the image that have high probability to attract a human observer s attention. Our system can easily adapt to multiple oveation points by changing the error sensitivity mask S(v,. Suppose that there are P oveation points x, x,, x in the image (in digitally sampled images, the oveation regions can also be regarded as collections o oveation P points). For each o the points, we can calculate the error sensitivity mask as in the above sections and have S i (v, or i =,,, P. The overall error sensitivity should be given by the maximum o them: S( v, = max( S ( v, ). (6) i = P i In practice, it is not necessary to compute each S i (v,. Since the error sensitivity is monotonically decreasing with increasing distance rom the oveation point, given a point x, the oveation point that is closest to it must generate the maximum S i (v,, so what we need to do is let S( v, S ( v, j arg min x x. (7) By doing this, a large amount o computation is saved. =, or j { i } i {,,, P} 4. IMAGE CODING USING THE FOVEATED QUALITY METRIC SPIHT [] is a very eicient progressive wavelet image coding algorithm. We designed a modiied SPIHT algorithm and tuned it using the above FWQI model to optimize the oveated visual quality at any given bit rate. We call the new coding algorithm the embedded wavelet image coding (EFIC) algorithm [5]. The encoded bitstream can be truncated at arbitrary place to create reconstructed images with dierent quality and depth o oveation. Fig. 4 shows the 5 5 Zelda image encoded with both SPIHT and EFIC algorithms. Fig. 5 gives the FWQI comparisons o the EFIC and SPIHT compressed Zelda images at , and 0.5bpp, respectively. The FWQI or each image is given as a unction o the viewing distance, instead o just one ixed value. Signiicant quality gain is achieved throughout the whole range o the viewing distances. This is consistent with the oveated subjective quality o Fig. 4. Fig. 6 shows how the FWQI result changes with the encoding bit rate. In Fig. 7, we compare the News image compression results at the same bit rate o 0.5bpp but with dierent ROI region selections. It turns out that uniorm resolution SPIHT coding cannot provide an acceptable image, but i the ROIs are known to us, visually satisactory quality image is still achievable with the EFIC algorithm. The EFIC decoding can also be viewed as a oveation iltering process with decreasing oveation depth. Note that, in typical natural images, the energy is concentrated in the low requency bands. As a result, in the peripheral regions, the low requency wavelet coeicients have greater opportunity to be reached beore the high requency ones. In the region o ixation, both low and high requency coeicients have good chances to be reached early because o their larger importance weights. I the bit rate is limited, then decoding corresponds to applying all-pass iltering to the region o ixation and low-pass iltering to the peripheral regions. This is the basic idea o oveation iltering. With an increase o the bit rate, more bits are received or the high requency coeicients o peripheral regions, thus the decoded image becomes less oveated. The EFIC coding results in Fig. 4 demonstrate this very well. 5. CONCLUSION AND DISCUSSION We described our oveated wavelet image quality measurement approach, which considers multiple actors o the HVS, including the space variance o the contrast sensitivity unction, the spatial variance o the local visual cut-o requency, the variance o human visual sensitivity in dierent wavelet subbands, and the inluence o the viewing distance on the display resolution and the HVS eatures. We show its eectiveness by using it as a guide or optimal bit assignment o an embedded oveated image coding system.

7

8 Fig.4 Zelda image compression result comparison. Top: Original image with the oveated ROI indicated; The let images that ollowed: SPIHT coded images; The right images that ollowed: EFIC coded images. The bit rates rom top to bottom are bpp (CR=5:), 0.035bpp (CR=56:), 0.065bpp (CR=8:), 0.5bpp (CR=64:), and 0.5bpp (CR=3:), respectively.

9 When we introduce our oveation image coding and processing work to people, the most requently asked question is: How do you know the oveation points? Generally, there are two methods to determine the ixation point(s) and region(s). The irst is a completely automatic method. There has been a lot o research work in the visual psychology community towards understanding high level and low level processes in deciding human ixation points [, 3]. High level processes involves a cognitive understanding o the image. For example, once a human ace is recognized in an image, the ace area is very likely to become a heavily ixated region. Low level processes determine the points o interest using simple local eatures o the image [3]. The second method to determine oveation point(s) is the interactive method. In some applications, an eye tracker is available, which can track the ixation point and send it to the oveated imaging system in real time. In some other application environments, the eye tracker is not available or inconvenient. A more practical way is to ask the users to indicate ixation points using a mouse. Another practical possibility is to ask the users to indicate the object o interest, and an automatic algorithm is employed used to track the user-selected object as the oveated region in the image sequence that ollows. It is worth noting that in a oveated system, no object segmentation is needed. As shown in Fig. 4 and Fig. 7, it is not necessary or a oveated system to extract the boundary o an object precisely. A rough oveated region is enough or the oveated system to work properly. Note that manually picking oveation points is much easier than manually deining ROIs in the image. Also note that automatically locating oveation regions is much easier than automatically segmenting objects rom the image. In this sense, a oveated image coding and communication system is more implementable, lexible, robust and thus practical than the segmentation-based ROI coding systems. REFERENCES [] L. R. Cormack, Computational models o early human vision, in Handbook o image and video processing, Al Bovik, Ed., Academic Press, May 000. [] W. S. Geisler, and J. S. Perry, A real-time oveated multiresolution system or low-bandwidth video communication, Proceedings o SPIE, vol. 399, 998. [3] E.-C. Chang, Foveation techniques and scheduling issues in thinwire visualization, Ph.D. Dissertation, New York University, May 998. [4] S. Lee, Foveated video compression and visual communication over wireless and wireline networks, Ph.D. Dissertation, The University o Texas at Austin, May 000. [5] Z. Wang and A. C. Bovik, Embedded oveation image coding, accepted by IEEE Trans. Image Processing, 00. [6] E. Atsumi, and N. Farvardin, Lossy/lossless region-o-interest image coding based on set partitioning in hierarchical trees, IEEE International Conerence on Image Processing, vol., pp. 87-9, 998. [7] J. Jung, S. Joung, Y. Jang, and J. Paik, Enhancement o region-o-interest coded images by using adaptive regularization, IEEE International Conerence on Consumer Electronics, pp. 6-63, 000. [8] T. L. Arnow, and W. S. Geisler, Visual detection ollowing retinal damage: predictions o an inhomogeneous retino-cortical model, Proceedings o SPIE: Human vision & Electro. Imaging, vol. 674, pp. 9-30, 996. [9] M. S. Banks, A. B. Sekuler, and S. J. Anderson, Peripheral spatial vision: limits imposed by optics, photoreceptors, and receptor pooling, Journal o the Optical Society o America, vol. 8, pp , 99. [0] A. B. Watson, G. Y. Yang, J. A. Solomon, and J. Villasenor, Visibility o wavelet quantization noise, IEEE Trans. Image Processing, vol. 6, no. 8, pp , Aug [] A. Said, and W. A. Pearlman, A new, ast, and eicient image codec based on set partitioning in hierarchical trees, IEEE Trans. Circuits and Systems or Video Technology, vol. 6, no. 3, pp , June 996. [] A. L. Yarbus, Eye Movements and Vision, Plenum press, New York, 967. [3] C. M. Privitera and L. W. Stark, Algorithm or deining visual regions-o-interest: comparison with eye ixations, IEEE Trans. Pattern Analysis and Machine Intelligence, vol., no. 8, pp , Sep. 000.

10 Fig. 5 FWQI comparison o EFIC and SPIHT compressed Zelda image at 0.565bpp, 0.065bpp and 0.5bpp. Fig. 6 FWQI results o EFIC compressed Zelda image at dierent bit rates.

11

12 Fig bpp (CR=3:) News image compression result comparison. Top-let: Original image with oveated ROIs indicated; Top-right: EFIC with the upper ROI only; Mid-let: EFIC with the lower let ROI only; Mid-right: EFIC with the lower right ROI only; Bottom-let: EFIC with all the three ROIs; Bottom-right: SPIHT uniorm resolution compression.