LOCAL MASKING IN NATURAL IMAGES MEASURED VIA A NEW TREE-STRUCTURED FORCED-CHOICE TECHNIQUE. Kedarnath P. Vilankar and Damon M.

Transcription

1 LOCAL MASKING IN NATURAL IMAGES MEASURED VIA A NEW TREE-STRUCTURED FORCED-CHOICE TECHNIQUE Kedarnath P. Vilankar and Damon M. Chandler Laboratory of Computational Perception and Image Quality (CPIQ) School of Electrical and Computer Engineering, Oklahoma State University, Stillwater, OK ABSTRACT It is widely known that natural images can hide or mask visual signals, and that this masking ability can vary across different regions of the image. Previous studies have quantified masking by measuring image-wide detection thresholds or local thresholds for select image regions; however, little effort has focused on measuring local thresholds across entire images so as to achieve ground-truth masking maps. Such maps could prove invaluable for testing and refining masking models; however, obtaining these maps requires a prohibitive number of trials using a traditional forced-choice procedure. Here, we present a tree-structured forced-choice procedure (TS-3AFC) designed to efficiently measure local thresholds across images. TS-3AFC requires fewer trials than normal forced-choice by employing recursive patch subdivision in which the child patches are not tested individually until the target is detectable in the parent patch. We show that TS- 3AFC can yield masking maps which demonstrate both intrasubject and inter-subject repeatability, and we analyze the performance of a modern masking model and two quality estimators in predicting the obtained ground-truth maps for a small setofimages. Index Terms Masking, visual detection, forced-choice procedure, image quality 1. INTRODUCTION Visual masking is a general term that refers to the ability of a masking image to reduce a subject s ability to visually detect a target signal placed upon the mask. Masking has found extensive use in image and video quality assessment, compression, watermarking, unequal error protection, image synthesis, and a variety of other applications. Accurately predicting the ability of an image to mask a given target is thus a crucial step in many image-processing and computer-graphic systems. One of the key challenges in designing a model of masking which is effective for natural images is the lack of groundtruth data, particularly local thresholds for full-sized images. The vast majority of masking models presented in the humanvision literature have been developed and refined to fit detection thresholds obtained using unnatural masks (e.g., sinewave gratings). More recent studies have measured detection thresholds for natural-image masks [1, 2, 3]; however, these thresholds were measured for full-sized images, and not for individual image regions. Indeed, different image regions can impose substantially different amounts of masking [4]. Forced-choice procedures are a reliable, objective, and widely accepted technique for measuring contrast detection thresholds [5]. Nonetheless, employing a force-choice procedure to measure local thresholds in full-sized images is impractical due to the time requirements. Consequently, studies which have investigated masking by natural images have yet to report local thresholds for entire images. For example, in [4], we employed a three-alternative forced-choice (3AFC) procedure to measure thresholds for detecting wavelet distortions in fourteen patches containing different structures; the time-consuming nature of the experiment prohibited us from measuring such thresholds for the patches of a full-sized image. [For example, measuring the local (64 64) thresholds in a image would have required us to test each of the image s sixty-four patches; over four times the number of patches tested in [4].] To address this issue, in this paper, we present a treestructured 3AFC testing paradigm (TS-3AFC) designed specifically to measure local thresholds in full-sized images. Rather than testing each fixed-size patch of the image individually, we employ dynamic patch sizes selected by recursively subdividing the patches of the image into quarter-sized subpatches (much like a dyadic wavelet transform). Specifically, suppose that the goal is to obtain contrast detection thresholds for each patch of a image. The TS-3AFC procedure starts with a single 3AFC trial for the full-sized image in which the target (e.g., wavelet distortion) has been applied to the entire image. Next, a single 3AFC sub-trial is conducted for each of the four patches of the image. For any of these patches in which the subject is unable to detect the target, the contrast of the target for the entire patch is increased. On the other hand, for each patch in which the subject correctly detects the target, the patch is further subdivided into four /11/$ IEEE 18 IVMSP 2011

2 patches, and then a sub-trial of the 3AFC procedure is conducted for each of these smaller patches. For any patch in which the subject cannot detect the target, the contrast of the target in the entire patch is increased. For any patch in which the subject successfully detects the target, the patch is further subdivided into four patches, and then a sub-trial of the 3AFC procedure is conducted for each of these smallest-size patches. Thus, during the experiment, the TS-3AFC procedure maintains a separate staircase track for each patch. During the second trial of the TS-3AFC procedure, the entire process is repeated, starting again with the full-sized image; this time, however, the contrast of the target spatially varies according to the results of the first trial. For those patches in which the target was undetectable, the contrast of the target has been increased relative to the first trial. For those patches in which the target was detectable, the contrast of the target has been decreased relative to the first trial. This process is repeated until the subject has conducted at least N trials on the full-sized image. The primary advantage of this technique is speed: The tree-structured nature of TS-3AFC requires fewer comparisons because the child patches are not tested individually until the target is detectable in the parent patch. Here, we demonstrate the efficacy of the TS-3AFC procedure for measuring local detection thresholds for wavelet distortions applied to a small set of natural images. By using N = 10 trials, we show that the TS-3AFC procedure can yield local detection thresholds, hereafter referred to as masking maps, which demonstrate both intra-subject and inter-subject repeatability. Subjects were able to reproduce their masking maps with an average correlation between maps of R =0.89; the masking maps for each image yielded an average correlation of R = 0.89 across different subjects. We also present the results of a quantitative analysis designed to test the accuracy of a mainstream computational model of masking [6] and two image quality estimators [7, 8] in predicting the masking maps. This paper is organized as follows: Section 2 describes the methods of the TS-3AFC procedure in the context of an experiment designed to measure masking maps for images containing wavelet distortion. Section 3.1 presents the results of the experiment and a discussion of the reliability of the procedure. Section 3.2 provides an analysis of the ability of various masking/quality algorithms in predicting the results. General conclusions are provided in Section METHODS: TS-3AFC APPLIED TO WAVELET DISTORTION DETECTION 2.1. Apparatus Stimuli were displayed on an HP LP246 monitor. The screen size of the monitor was 24 inches at a display resolution of pixels/cm and a frame rate of 59.7 Hz. The display yielded minimum, maximum and mean luminance of 0.38, 350, and 76.5 cd/m 2, respectively, and overall gamma of 2.2. Stimuli were viewed binocularly through natural pupils in a darkened room at a distance of approximately 60 cm Stimuli Stimuli were generated using four commonplace images (balloon, horse, lena, andgirl; see Figure 4 at the end of this paper). The images were of size pixels and were 8-bit grayscale with pixel values in the range Each stimulus was composed of a target and a mask. Masks consisted of the original images and targets consisted of distortions generated by adding Gaussian noise to the wavelet subbands of the image. Specifically, a discrete wavelet transform was applied to each original image using the 9/7 biorthogonal filters and three decomposition levels. Noise was added into the vertical (HL) subband at the third decomposition level. The subband was divided into 64 equalsized blocks in which each block of coefficients corresponded to an approximately spatial region in the image. The standard deviation of the noise was varied for each block of coefficients as specified by the staircase procedure, thus resulting in a variable amount of noise added to each spatial region. At the display visual resolution of pixels/degree, the HL subbands at the third level of decomposition corresponded to a center spatial frequency of 4.3 cycles/degree Procedure Thresholds were measured by using a spatial three alternative forced-choice procedure. On each trial, subjects concurrently viewed three adjacent images placed on a uniform 76.5 cd/m 2 background. Two of the images contained the mask alone, and the other image additionally contained one of the previously described targets (distortions). Subjects indicated by means of keyboard input which of the three images contained the target. During each trial, an auditory tone indicated stimulus onset, and auditory feedback was provided on an incorrect response. Subjects were instructed to examine all three images before responding. Response time was limited to within 5 seconds of stimulus onset, during which all three images remained visible. The images shown to the subjects were not of same dimensions during each trial. The dimensions of the three images varied from trial to trial, ranging from the fullsized image dimensions of to the smallest patch size of As described next, the dimensions of the three images on the next trial depended on the subject s previous responses. Thresholds were defined as the RMS contrast of the distortion generated by adding Gaussian noise with a standard deviation given by the mean of the last six trials of the staircase procedure for each block. 19

3 2.4. Tree-Structured 3AFC Procedure The staircase procedure used in this experiment had 10 main trials. The first main trial starts with the same amount of distortion in each block of the image. Subsequent main trials of the experiment use spatially varying amounts of distortion based on the results of previous trials. Each main trial consists of a variable number of sub-trials which follow a depth-first search to traverse the tree of blocks (refer to Figure 1 for the following discussion). The first sub-trial starts at Level 1 in which the full-sized ( ) image is displayed to the subject. If the first subtrial s response is correct, then the next four sub-trials are of the Level 2 patches which are a quarter of the image ( ). However, if the first sub-trial s response is incorrect, then the amount of distortion in each patch is incremented by a fixed amount, and then the experiment goes to the next main trial. The next main trial starts with the first sub-trial at Level 1 and an updated amount of distortion modified according to the previous main trial for each patch. Level 2 contains four sub-trials, each displaying only a quarter ( ) of the original image. If the subject s response for a Level 2 sub-trial is correct, then the patch in that sub-trial goes into a further four sub-trials at Level 3 where each quarter ( ) of the patch in Level 2 is displayed. However, if the subject s response to a sub-trial at Level 2 is incorrect, then the patch in that trial does not go into further sub-trials of Level 3; instead, the amount of distortion in the entire patch is incremented by a fixed amount. Similarly at Level 3, if the subject s response for a subtrial is correct, then the patch in that sub-trial goes to four further sub-trials at Level 4 using patches. However, if the subject s response for a Level-3 sub-trial is incorrect, then the amount of distortion in the entire patch is incremented. Sub-trials of Level 4 display patches of dimensions (the smallest size tested). At this level, if the subject s response is correct, then the amount of distortion in the patch is decremented by a fixed amount. If the subject s response is incorrect, then the amount of distortion is incremented by a fixed amount. Figure 1 shows an example of the experimental procedure of a main trial and all the sub-trials of that main trial. For illustration, the image is divided into blocks of size The blocks are represented by numbers in the figure; each number indicates the position of the block in the image. In this figure, C indicates a correct response and IC indicates an incorrect response. In first sub-trial (Level 1), the full-sized image is displayed. The subject gives the correct response, hence the image goes to Level 2 s subtrials. At Level 2, the subject is displayed four sub-trials with the four patches. For some of these patches, the subject gives an incorrect response. These patches do not go Fig. 1. Example of the TS-3AFC procedure. This figure shows the all the sub-trials at each level of a main trial in the experiment. C indicates a correct response from the subject, and IC indicates an incorrect response. into next level of sub-trials; instead, the amount of distortion in these patches is incremented for the next main trial. The Level-2 patches which get correct responses go to Level 3 s sub-trials. This process is repeated at Level 3 where the subject performs four sub-trials on the patches of the Level-2 patches which yielded a correct response. Finally, at Level 4, the patches are displayed, and the amount of distortion is adjusted up or down based on the response Observers Six adult subjects participated in the experiment, including the authors. Five of the observers were experienced with near-threshold distortion detection experiments. One of the subjects (NS) was naive to the purpose of the experiment and had no previous experience in visual detection tasks. All had either normal or corrected-to-normal visual acuity. Each subject performed the experiment on one image, with the exception of the first author (KV) who performed the experiment on three images. To measure intra-subject 20

4 Table 1. Intra- and inter-subject correlation between threshold maps obtained from the experiment. Intra Inter Subject s Image Subject Subject Initial name Correlation Correlation KV balloon TP balloon 0.93 KV horse PV horse 0.89 CV lena DC lena 0.85 KV girl NS girl 0.90 repeatability, each subject took the experiment twice on the same image(s); the second experimental session was performed on a different day, where the time interval between the two sessions ranged from 1-6 days. To measure intersubject repeatability, each image was viewed by at least two subjects. 3. RESULTS AND ANALYSIS Figure 4 (located at the end of this paper) shows the resulting masking maps (local RMS contrast thresholds, averaged across subjects and sessions), for each of the four images. The leftmost column shows the original images, and the second column shows the corresponding maps. For display purposes, these maps have been normalized according to the scale shown on the right-hand portion of the figure. The ground-truth maps clearly demonstrate the variation in masking across different spatial regions. In general, textured regions demonstrate the greatest amount of masking; smooth regions demonstrate the least amount of masking. For example, the map for image balloon shows that the thresholds are higher in the ground as compared to the sky. Also the edges of the balloon impose some masking, but much less than the rough ground. Similarly, in the map for image horse, the ground and horse s fur can hide more than the sky and upper branches. In the map for lena, the feathers and Lena s hair can hide distortions than her smooth skin. In the map for image girl, the texture of the girl s sweater and hair hide more distortion than the face and smooth background Reliability of the method The TS-3AFC procedure can yield thresholds which are repeatable. As shown in Table 1, the intra-subject correlation between thresholds from a single subject across different sessions was on average R = The inter-subject correla- Fig. 2. RMS contrast thresholds averaged across sessions and subjects for each image. Each error bar represents one standard deviation of the respective mean. tion between thresholds from different subjects on the same image was on average R = Thus, subjects were generally able to reproduce their thresholds across sessions, and they were able to achieve thresholds which are consistent with other subjects. Although the time interval between the sessions varied greatly, the lowest intra-subject correlation was 0.85 by subject DC (the second author). The naive subject, NS, achieved a correlation of 0.9. To further demonstrate the repeatability of the maps across sessions and subjects, Figure 2 shows the average thresholds and corresponding error bars (one standard deviation of the mean across subjects and sessions) for each of the four images. Figure 3 shows the per-session, per-subject masking maps for subjects KV and NS on image girl. 21

5 Fig. 3. Per-session, per-subject masking maps for subjects KV and NS on image girl, demonstrating the repeatability of the maps across sessions and subjects Predicting the masking maps Finally, we analyze the performance of a mainstream computational model of masking and two image quality estimators in predicting the masking maps. We tested the computational neural model of Watson and Solomon [6], MSSIM [7], and MAD [8]. Each of these algorithms can take as input an original image patch and a distorted image patch, and yield as output a scalar index corresponding to quality/distortion/visibility. We selected a fixed index for each algorithm corresponding to an at-threshold amount of distortion. Then, for each block of the image, we successively added wavelet distortion until the algorithm yielded the atthreshold index for that block. Finally, the resulting masking map was computed as the RMS contrast of the distortion in each of these distorted blocks. Figure 4 shows the predicted masking maps. In general, the maps predicted by the models are similar to the groundtruth maps. However, Watson s model and MSSIM overpredict the thresholds in some regions (see, e.g., the thresholds for the tree branches in the top-right corner of image horse). The thresholds predicted by MAD are generally too conservative. For example in image girl, MAD underestimates the masking ability of the girl s sweater. Table 2 shows the performance of local detection threshold maps obtained from the experiment and maps from other algorithms on the four images. We used five criteria to compare the performances: (1) Pearson correlation coefficient (CC), which measures how well an algorithms predictions correlate with the subjective scores; (2) Spearman rank-order Table 2. Overall performance of local detection threshold maps obtained from the experiment and maps from other algorithms on the four images. balloon horse lena girl Watson s CC SROCC OR OD KLD MSSIM CC SROCC OR OD KLD MAD CC SROCC OR OD KLD correlation (SROCC), which measures the relative monotonicity between the predictions and subjective scores; (3) and (4) the outlier ratio, OR, and outlier distance, OD, both of which attempt to account for the inherent variation in human subjective ratings of quality. The Kullback Leibler divergence (KLD) measures the difference between two probability density functions estimated from the histograms of the maps. All of these measures were computed after applying a best-fitting sigmoid transformation of the algorithm outputs. Overall, the model of Watson & Solomon and MSSIM provide the best predictions of ground-truth, with correlations nearing those of the humans across sessions and subjects. However, we have tested only four images; clearly, many more images are needed before one can draw definitive conclusions. 4. CONCLUSION This paper presented a tree-structured forced-choice procedure (TS-3AFC) designed to efficiently measure local thresholds across images. By employing recursive patch subdivision in which the child patches are not tested individually until the target is detectable in the parent patch, the TS-3AFC procedure requires fewer trials than a traditional forced-choice procedure. When applied to the task of detection wavelet distortions in a small suite of test images, we showed that TS-3AFC can yield masking maps which demonstrate both intra-subject and inter-subject repeatability. Clearly, further testing on a larger database is required to fully test the reliability of the method. We are currently in the process of collecting these data using images from the CSIQ 22

6 Fig. 4. Original images, ground-truth masking maps, and predicted masking maps. The X in the maps for image horse denote a block that was too dark to measure a contrast threshold (luminance less than 3 cd/m 2 ). image quality database [8], which will allow us to examine the relationship between masking maps to quality ratings. 5. REFERENCES [1] T. Caelli and G. Moraglia, On the detection of signals embedded in natural scenes, Perception and Psychophysics, vol. 39, pp , [2] M. J. Nadenau and J. Reichel, Image-compressionrelated contrast-masking measurements, in Human Vision and Electronic Imaging V,B.E.RogowitzandT.N. Pappas, Eds., 2000, vol. 3959, pp [3] D. M. Chandler and S. S. Hemami, Effects of natural images on the detectability of simple and compound wavelet subband quantization distortions, J. Opt. Soc. Am. A,vol. 20, no. 7, July [5] N. Graham, Visual Pattern Analyzers, Oxford University Press, New York, [6] A. B. Watson and J. A. Solomon, A model of visual contrast gain control and pattern masking, J. Opt. Soc. Am. A, vol. 14, pp , [7] Z. Wang, E. P. Simoncelli, and A. C. Bovik, Multi-scale structural similarity for image quality assessment, in Proc. Asilomar Conf. on Signals, Systems, and Computers, November [8] Eric C. Larson and Damon M. Chandler, Most apparent distortion: full-reference image quality assessment and theroleofstrategy, Journal of Electronic Imaging, vol. 19, no. 1, pp , [4] Damon M. Chandler, Matthew D. Gaubatz, and Sheila S. Hemami, A patch-based structural masking model with an application to compression, J. Image Video Process., vol. 2009, pp. 1 22,