Geometrical and Statistical Properties of Vision Models obtained via Maximum Differentiation

Transcription

1 To appear in: Proc. SPIE, Conf. on Human Vision and Eectronic Imaging, XX Vo. 9394, Eds. B Rogowitz, TN Pappas, H de Ridder, San Francisco, CA, 9-12 Feb Geometrica and Statistica Properties of Vision Modes obtained via Maximum Differentiation Jesús Mao and Eero P. Simoncei jesus.mao@uv.es, ABSTRACT We examine properties of perceptua image distortion modes, computed as the mean squared error in the response of a 2-stage cascaded image transformation. Each stage in the cascade is composed of a inear transformation, foowed by a oca noninear normaization operation. We consider two such modes. For the first, the structure of the inear transformations is chosen according to perceptua criteria: a center-surround fiter that extracts oca contrast, and a fiter designed to seect visuay reevant contrast according to the Standard Spatia Observer. For the second, the inear transformations are chosen based on statistica criterion, so as to eiminate correations estimated from responses to a set of natura images. For both modes, the parameters that govern the scae of the inear fiters and the properties of the noninear normaization operation, are chosen to achieve minima/maxima subjective discriminabiity of pairs of images that have been optimized to minimize/maximize the mode, respectivey (we refer to this as MAximum Differentiation, or MAD, Optimization). We find that both representations substantiay reduce redundancy (mutua information), with a arger reduction occurring in the second (statisticay optimized) mode. We aso find that both modes are highy correated with subjective scores from the TID2008 database, with sighty better performance seen in the first (perceptuay chosen) mode. Finay, we use a foveated version of the perceptua mode to synthesize visua metamers. Specificay, we generate an exampe of a distorted image that is optimized so as to minimize the perceptua error over receptive fieds that scae with eccentricity, demonstrating that the errors are barey visibe despite a substantia MSE reative to the origina image. Keywords: Vision Modes, Muti-ayer Networks, Image Quaity Metrics, Maximum Differentiation, Redundancy Reduction, Visua Metamers. Introduction It is widey beieved that visua perception emerges through a cascaded sequence of neura transformations having a simiar canonica form. 1 4 It is thus natura to construct modes for image quaity assessment using cascades, in which each stage anayzes factors of increasing compexity (e.g. uminance, contrast, and structure). 5 7 Critica to this endeavor is the probem of parameter optimization: How can we seect parameters of a muti-stage cascade mode so as to mimic human discrimination performance? Recent progress in object recognition has expoited such cascaded constructions, aong with new methods of supervised machine earning appied to arge databases of natura images, to achieve state-of-the-art resuts [e.g., 8 10]. But these Image Processing Lab, Universitat de Vaència, Spain Howard Hughes Medica Institute, Center for Neura Science, and Courant Institute of Mathematica Sciences, New York University, USA This work was partiay supported by the projects TIN C03-01 and TIN exp.

2 machine earning methods rey on data sets (images, and category abes) that are many orders of magnitude arger than what can be feasiby obtained in experiments with human subjects. Here, we assume a cascade mode in which each stage is a inear-noninear transformation. The inear part is a convoution with one or more fiters, and the noninear part impements a oca divisive normaization operation (see 11 for review). Such representations have been proposed previousy for perceptua image distortion measures, 5, 7, 12, 13 as we as other image processing appications. 14 We use perceptuay or statisticay motivated choices of the fiters for each stage. 7, 13 Given these fiters, we determine the remaining scaar parameters of the normaization operations based on human subject responses in a Maximum Differentiation (MAD) task. 15 Specificay, subjects examine pairs of images synthesized with maxima/minima distortion according to modes with different parameter settings, and choose the image pairs that are most easiy differentiated. We refer to this as MAD optimization. Preiminary resuts ead to modes in which each transformation step provides a successivey more powerfu measure of perceptua quaity, as measured by its abiity to synthesize images with minimay visibe distortion but with substantia mean squared error. Moreover, each step reduces the statistica redundancy (mutua information) of natura images. Both modes are highy correated with human quaity scores from the TID2008 database. 16 Finay, we show that a foveated version of the perceptuay-optimized mode, in which fiters grow in diameter with eccentricity (as in the eary visua system), can be used to synthesize images with minimay visibe distortions of even arger mean squared error. Structure of cascaded modes We assume a mode constructed as a cascade of stages: x (0) T 1 x (1) T 2 x (2) (1) where x (0) is a pixeated input (uminance) image containing d pixes, each T i performs a inear+noninear transformation on the vector x (i) to give the set of responses in the vector x (i+1). The quaity measure is simpy the MSE measured on the output ayer, x (2). The high dimensionaity of the signas can impy a arge number of parameters in each transformation T i. For instance, a genera inear operation preserving the dimensionaity in the i-th stage is parameterized by a matrix with d d free parameters. The number of parameters in the noninear part may be even bigger. In order to keep the number of parameters sma, the structure of the stages has to be constrained. We assume the inear stages are convoutiona, 17 and choose the fiters according to either perceptua or statistica criterion. The resuting transforms are somewhat simiar, as suggested 13, 18, 19 by the Efficient Coding Hypothesis. Perceptua design The fiters are chosen using we-known conventions in the perception and image quaity communities. The first stage computes the oca contrast by subtracting and normaizing by the oca uminance (computed with a Gaussian kerne of width σ 1 ): x (1) k = ( I H (1) k ) x (0) b 1 + H (1) k x (0) where I is the identity matrix. The second stage appies a inear Contrast Sensitivity fiter and a masking noninearity: x (2) k = sign(h CSF k x (1) ) Hk CSF x (1) γ2 Hm CSF b 2 + K (2) k (2) (3) x(1) m γ2

3 where H CSF is the convoution by the impuse response corresponding to the Contrast Sensitivity Function of the Standard Spatia Observer CSF. 20 Overa, we just have two parameters in the first stage (σ 1, b 1 ) and three parameters in the second stage (σ 2, b 2 and γ 2 ). Statistica design. We set the inear part, H (i), to be a decorreating transform (PCA), over a database of natura images. 21 On the other hand, we assume the noninear part is 13, 19, 22, 23 a divisive normaization, which has been shown to reduce redundancy: x (i) k = D(i) kk sign(h(i) k x (i 1) ) H (i) k x (i 1) γi b k + K (i) k (4) H (i) m x(i 1) m γi The normaization pooing, corresponding to K (i), is computed with a Gaussian bur of width σ i,andd (i) is a diagona matrix containing a weighting function with an exponentia decay in the diagona, with characteristic ength κ i (i.e., it appies arger weights to the first coefficients of the decomposition defined by H (i) ). Since H (i) is chosen based on image statistics (it is the eigenvector matrix of the covariance of sampes at the (i 1)-th stage), there are ony four unknowns per stage: the width, σ i, the scaar parameters b i and γ i, and the width κ i of the post-weighting. Perceptua optimization of normaization parameters To obtain perceptua estimates of the scaar parameters of either mode, we used the Maximum Differentiation (MAD) competition methodoogy. 15 Specificay, for a given set of parameters, we synthesize images that are maxima/minima in the MSE of their mode responses reative to those of an origina image, but have the same MSE in the image domain (again, reative to the origina image). Synthesis is achieved as foows. First we add white noise to the origina image to achieve a desired pixe-domain Eucidean distance (MSE). Then, we modify this initia image through gradient ascent/descent on the MSE of the mode response, whie constraining the image to ie on the sphere of desired image-domain MSE. The gradient is computed by concatenating the Jacobian matrices corresponding to each stage of the transformation (i.e., the chain rue ). We aso impemented a more efficient cacuation based on a a second-order approximation 7, 13 of the mode response distance. Under this approximation, MAD competition reduces to injecting noise into the subspace corresponding to ow/high eigenvaues of the transformed Riemannian metric. This subspace describes the east/most visibe directions according to the mode. Interestingy, we observed that even for image-domain PSNR 25, this second order approximation of the mode response distance, 7, 13 produces very simiar resuts to the fu gradient-descent optimization. According to the mode with the seected parameter settings, the two synthesized images are deemed the most perceptuay different pair of images at that eve of (image-domain) MSE. By comparing image pairs synthesized for different parameter settings, a human observer can seect those that are most distinguishabe, thereby indicating the choice of parameters that agrees best with their perceptua judgements. We deveoped an iterative psychophysica coordinate descent procedure to estimate the scaar normaization parameters for both modes. In an outer oop, we aternated between optimizing the parameters of the first stage (initiay, assuming an unnormaized transformation for the second stage), and then optimizing those of the second stage. Within each stage, we optimized one parameter at a time, hoding the others fixed at their current vaues. MAD

4 optimization of one parameter was achieved by showing observers pairs of extrema images synthesized with six different vaues of that parameter (but with the same image-domain MSE) and asking them ro specify which pair exhibited the argest difference in quaity. After each tria, new pairs were drawn, keeping the seected parameter vaue from the previous tria, but using six different vaues of a different parameter. A staircase procedure in which the ranges of the parameters were progressivey reduced was used to search for the optima vaue of each parameter. This was repeated for mutipe initia conditions and observers and the optima averaged to obtain an overa estimate for the parameters. We ooped over the set of parameters four times when optimizing each stage, and we iterated over the two stages unti stabe resuts (within the standard deviation of the parameters) were obtained. In the expored cases, this ony required two or three iterations. Experimenta data were coected for two subjects, and origina images were drawn from randomy chosen patches of standard images (Barbara, Einstein, Baboon, Cameraman, Boats, Godhi). Perceptua properties: Differentiation of cascade stages Figure 1 demonstrates the perceptua capabiities achieved by each mode stage, for both modes. Specificay, each row shows sequences of images with identica MSE in the image domain, but with minima MSE at at each successive stage of the corresponding mode cascade. As expected, the visibiity of errors in the perceptua mode are progressivey ess noticeabe, since the mode stages are optimized to represent perceptua distortion. More surprising is that this approximatey hods for the statistica mode, despite the fact that its inear stages are optimized soey to decorreate their inputs, as derived from natura images. Statistica properties: Redundancy reduction Tabe 1 demonstrates the reduction of statistica redundancy achieved by successive mode stages, for both modes. Specificay, we measured mutua information (in bits) between one coefficient and 8 neighbors at different stages of the network. Specificay, we used the eight adjacent coefficients of 3 3 spatia neighborhoods for the perceptua stages, and the eight adjacent frequencies in DCT-ike domains arising in the statistica stages. We averaged these vaues over different coefficients and across a set of patches of size from natura images from a caibrated database. 21 For both modes, we see a significant reduction in redundancy. Such a reduction is expected for the statisticay optimized mode, but consistent with ref., 19 it is aso found in the mode that is perceptuay derived, suggesting that it is a property of the eary visua system. Mode \ Stage Input Linear 1 Non-inear 1 Linear 2 Non-inear 2 Perceptua Statistica Tabe 1. Mutua information (in bits) between groups of neighboring coefficients at intermediate stages of each hierarchica mode. Properties of the distance: Comparison to subjective quaity ratings We compared image quaity predictions of our two modes to human subject data from the TID2008 database. 16 Figure 2 shows the aignment between subjects Differentia Mean Opinion Scores (DMOS) and distances computed according to our MAD-optimized modes (in bue) as

5 Figure 1. Exampe images synthesized to have minima distortion according to successive mode stages, but identica MSE in the image domain (PSNR = 28.1). The samping frequency used in the exampe impies that images shoud subtend 1 deg. Left to right: output of first inear transform, output of first normaization transform, output of second inear transform, output of fina normaization transform. Top: perceptua mode. Bottom: statistica mode. The distortion associated with one-ayer inear modes (eft) is simiar to white noise because the corresponding metric is stricty the identity in the statistica case and not far from the identity in the perceptua case.

6 Figure 2. Direct comparison of subjective image distortion data (DMOS) and distances computed according to different distortion measures: our MAD-optimized modes (scatter pots in bue) as we as severa modes from the iterature (red - from upper eft: RMSE, SSIM, 5 VIF, 25 MSSIM, 24 divnorm, 7 FSIM 26 ). The vaues of the Pearson, Spearman and Kenda correations (r p, r s and r k,respectivey) are provided above each pot. The vaue of the RMSE of the best inear fit (in DMOS units), from which r p is computed, is provided underneath each pot. we as those of severa other distortion measures (in red). We find that both modes show higher correation with the human data than SSIM, 5 MSSIM, 24 VIF 25 as we as those based on divisive normaization on waveets. 7, 12 Moreover, it provides a more inear scatter pot and more baanced residuas than the current state-of-the-art FSIM method. 26 Perceptua properties: Synthesis of foveated visua metamers The modes proposed here are both based on spatiay oca computation. In the eary visua system, the receptive fieds of neurons are ocaized, but grow approximatey ineary with eccentricity away from the fovea. This suggests that a better mode might be achieved by scaing the spatia fiters with eccentricity. A reated approach has shown that a oca representation of visua texture, when suitaby scaed with eccentricity, can generate highy distorted versions of images that are perceptuay indistinguishabe from an origina, when viewed under proper 27, 28 fixation. To demonstrate this concept, we modified the perceptua mode by introducing an eccentricity dependency in the widths of a the convoutiona kernes in Eqs. 2-3 whie keeping the other parameters constant. In particuar, for eccentricity smaer than 1 degree, the impuse response widths were those found in the MAD experiment. For bigger eccentricities, widths were ineary

7 Figure 3. Minimay visibe distortion, generated using a foveated version of the perceptua mode. The samping frequency used in the exampe impies that images shoud subtend 8 deg. Top row show images, and bottom row shows the difference between that image and the origina. Left: origina image. Midde: origina image corrupted by homogeneous white noise. Right: origina image corrupted by noise that is minimay visibe, according to the foveated perceptua mode. Mean squared error of the two corrupted images is identica (image-domain PSNR = 28.1). increased from their origina vaues at a rate of 0.06 subtended degrees per eccentricity degree. Then using the gradient-descent MAD procedure described above, we synthesized an image with noise that is barey visibe according to this foveated mode. Figure 3 shows an exampe, in comparison to an image corrupted by white noise of the same average ampitude. Note that the perceptua mode injects noise that is moduated over both space and spatia frequency, depending on a combination of eccentricity and oca image content. Concusion We ve deveoped an iterative psychophysica methodoogy, MAD-optimization, for seecting the parameters of perceptua distortion modes, and have used it to optimize two exampe modes. Both optimized modes show successive reductions in statistica redundancy, as we as reductions in the visibiity of distortions that they deem minima, over each mode transformation. And both modes show high correation with human quaity ratings. Armed with this optimization methodoogy, we hope to extend these modes with additiona stages, thus mimicking the hierarchica structure of the human visua system.

8 REFERENCES [1] Fukushima, K., Neocognitron: A sef-organizing neura network mode for a mechanism of pattern recognition unaffected by shift in position, Bioogica cybernetics 36(4), (1980). [2] Dougas, R. J., Martin, K. A., and Whitteridge, D., A canonica microcircuit for neocortex, Neura Computation 1, (1989). [3] Heeger, D. J., Simoncei, E. P., and Movshon, J. A., Computationa modes of cortica visua processing, Proc. Nat Academy of Science 93, (January 1996). [4] Riesenhuber, M. and Poggio, T., Hierarchica modes of object recognition in cortex, Nature Neuroscience 2, (1999). [5] Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncei, E. P., Image quaity assessment: from error visibiity to structura simiarity, Image Processing, IEEE Transactions on 13, (Apr. 2004). [6] Seshadrinathan, K. and Bovik, A. C., Unifying anaysis of fu reference image quaity assessment., in [ICIP], , IEEE (2008). [7] Laparra, V., Muñoz Marí, J., and Mao, J., Divisive normaization image quaity metric revisited, JOSA A 27(4), (2010). [8] Ranzato, M., Huang, F., Boureau, Y., and LeCun, Y., Unsupervised earning of invariant feature hierarchies with appications to object recognition, in [Proc. Computer Vision and Pattern Recognition Conference (CVPR 07)], IEEE Press (2007). [9] Jarrett, K., Kavukcuogu, K., Ranzato, M., and LeCun, Y., What is the best mutistage architecture for object recognition?, in [Proc. Internationa Conference on Computer Vision (ICCV 09)], IEEE (2009). [10] Saakhutdinov, R. and Hinton, G., An Efficient Learning Procedure for Deep Botzmann Machines, Neura Computation 24, (Apr. 2012). [11] Carandini, M. and Heeger, D. J., Normaization as a canonica neura computation., Nature reviews. Neuroscience 13, (Jan. 2012). [12] Teo, P. and Heeger, D., Perceptua image distortion, Proceedings of the SPIE 2179, (1994). [13] Mao, J., Epifanio, I., Navarro, R., and Simoncei, E. P., Non-inear image representation for efficient perceptua coding, IEEE Trans Image Processing 15, (Jan 2006). [14] Lyu, S. and Simoncei, E. P., Statisticay and perceptuay motivated noninear image representation, in [Proc. SPIE, Conf. on Human Vision and Eectronic Imaging XII], Rogowitz, B., Pappas, T. N., and Day, S. J., eds., 6492, 67 91, Society of Photo-Optica Instrumentation, San Jose, CA (January ). [15] Wang, Z. and Simoncei, E. P., Maximum differentiation (MAD) competition: A methodoogy for comparing computationa modes of perceptua discriminabiity, Journa of Vision 8, 1 13 (Sep 2008). [16] Ponomarenko, N., Cari, M., Lukin, V., Egiazarian, K., Astoa, J., and Battisti, F., Coor image database for evauation of image quaity metrics, Proc. Int. Workshop on Mutimedia Signa Processing, (Oct. 2008). [17] LeCun, Y. and Bengio, Y., Convoutiona networks for images, speech, and time-series, in [The Handbook of Brain Theory and Neura Networks], Arbib, M. A., ed., MIT Press (1995). [18] Barow, H., Possibe principes underying the transformation of sensory messages, in [Sensory Communication], Rosenbith, W., ed., , MIT Press, Cambridge, MA (1961).

9 [19] Mao, J. and Laparra, V., Psychophysicay tuned divisive normaization approximatey factorizes the PDF of natura images, Neura Computation 22(12), (2010). [20] Watson, A. and Mao, J., Video quaity measures based on the standard spatia observer, Proc. IEEE ICIP 3, (2002). [21] van Hateren, J. and van der Schaaf, A., Independent component fiters of natura images compared with simpe ces in primary visua cortex, Proc.R.Soc.Lond. B 265, (1998). [22] Schwartz, O. and Simoncei, E., Natura signa statistics and sensory gain contro, Nat. Neurosci. 4(8), (2001). [23] Lyu, S. and Simoncei, E. P., Noninear extraction of independent components of natura images using radia Gaussianization, Neura Computation 21, (Jun 2009). [24] Wang, Z., Simoncei, E. P., and Bovik, A. C., Muti-scae structura simiarity for image quaity assessment, in [in Proc. IEEE Asiomar Conf. on Signas, Systems, and Computers, (Asiomar], (2003). [25] Sheikh, H., Bovik, A., and de Veciana, G., An information fideity criterion for image quaity assessment using natura scene statistics, IEEE Transactions on Image Processing 14, (Dec 2005). [26] Zhang, L., Zhang, L., Mou, X., and Zhang, D., Fsim: A feature simiarity index for image quaity assessment., IEEE Transactions on Image Processing 20(8), (2011). [27] Rosenhotz, R., What your visua system sees where you are not ooking, in [Proc. SPIE: Human Vision and Eectronic Imaging], XVI (2011). [28] Freeman, J. and Simoncei, E. P., Metamers of the ventra stream, Nature Neuroscience 14, (Sep 2011).