An Implementation of Spatial Algorithm to Estimate the Focus Map from a Single Image

Transcription

1 An Implementation of Spatial Algorithm to Estimate the Focus Map from a Single Image Yen-Bor Lin and Chung-Ping Young Department of Computer Science and Information Engineering National Cheng Kung University Tainan City, Taiwan(R.O.C.) yen bor@yahoo.com.tw, cpyoung@mail.ncku.edu.tw Abstract In this paper, the implementation to estimate the focus map spatially based on the intentional reblur of one image, which is the only input data, is presented. This enables flexible computation in the spatial domain, rather than the frequency domain. The gradient magnitude term widely used in image processing was used to derive a ratio map. The pixels closer to the focal point of the camera were on the thinner edge in the proposed ratio map. Oppositely, The edge was wider the when the pixels were far from the focal point. The core of proposed algorithm is introduced and the experimental results demonstrate the enhanced effectiveness, quality, and precision of the proposed algorithm. Index Terms image processing, focus map, visualization. I. INTRODUCTION Stereo vision is evolving, and 3D-display technology and autostereoscopic displayers are being sold commercially. A displayer requires 3D information to show a scene and objects on a screen. Depth image-based rendering (DIBR) is a popular and practical algorithm in which the depth map is used as the source of 3D information. When converting 2D to 3D video, the quality of the depth map affects the rendering quality. However, the technique used to produce new material or transform pre-existing, conventional 2D content into a new format for 3D devices requires additional research. Application of the depth map includes deblur [1], enhancing images [2], performing artificial tasks, post-exposure refocusing and defocusing [3], automatic scene segmentation, object tracking, re-rendering of the scene from an alternative viewpoint, and recovering the 3D shape from a 2D image. A depth map is an 8-bit gray-level image. The pixel value is zero in objects farthest from the camera and 255 in objects closest to the camera. Depth information can be sourced from hardware equipment, such as scanners, or it can be estimated from various software-based algorithms. The focus map is another representation that is similar to, and often confused with, the depth map. The two terms have been used interchangeably in some articles, but depth and focus maps are not the same in theory or in application. Like depth maps, focus maps are 8-bit gray-level images, but they have a pixel value of zero in the most focused objects and a value of 255 in the most unfocused objects. The focus map is similar to the depth map when the focused object is located closest to the camera. It may be possible to use an algorithm to identify one map by using another map. To translate the two kinds of maps, a problem that must be solved is that the same value in a focus map is given for the objects that are at an equal distance to the focal plane. This value provides no information on whether a point is between the focal plane and the camera, or behind the focal plane. In other words, no information is given to determine which object is in front. However, the focal point is usually the location of the region of interest (ROI). This paper introduces the implementation of the proposed spatial algorithm to estimate the focus map from a single image. Focus and depth maps can be created using either the singleframe method, in which one image is used, or the multi-frame method. A previous study provided an introduction to depth and focus maps [4]. The purpose of this study is to design an efficient and high-quality algorithm for extracting the focus map from a single image captured with conventional, singleview equipment without any extra hardware. II. THE PROPOSED ALGORITHM This algorithm is used to estimate the focus map by inputting a single image. The focus map is denoted as D full, which is shown in (1) and expected to be derived from spatial variables and operations as described in the following paragraphs. D full (x, y) =d u (x, y) c (1) where c is a constant and D full is a relative value with respect to d u that denotes the distance between the object and the focal plane. The ratio of original gradient magnitude to intentionally reblurred gradient magnitude is always an intermediary computation term in the image processing techniques. However, it is possible to derive the focus information according to the observation on the proposed ratio map in this paper, as shown in Fig. 1(a), by eliminating the infinite value and visualizing the rearranged intermediary data. The method of estimating the focus information will be introduced. The edges of the original input image were blurred extensively using an intentional reblur, which caused different edge widths to appear on the ratio map. According to observation, the widths of the edges on the ratio map were found to be related to the distances of the objects from camera focus. The pixels of the object which is closer to the focal point of the camera are on the thinner edge. The edge is wider when the pixels of the object are far /16/$31.00 copyright 2016 IEEE ICIS 2016, June 26-29, 2016, Okayama, Japan

2 from the focal point. The detailed flowchart, which shows the steps of the proposed algorithm, is provided in Fig. 2. (a) The original input (b) The gradient magnitude of original image (a) The ratio map (b) The focus-distance edge map Fig. 1. A ratio map and the corresponding focus-distance edge map. 1) Intentional Reblur: The popular Gaussian filter MASK Gaussian defined in (2) was applied to the grey image to reblur the data. Input data was intentionally blurred again to estimate the degree of the original blur. The reblurred image is denoted as H, and a convolution was applied using a Gaussian filter, as shown in (3). This step is a key feature of this algorithm. The degree and the type of original blur are both unknown when the image is taken. The parameters of the reblur are known in advance. MASK Gaussian (x, y) = 1 2πσ 2 x 2 +y 2 e 2σ 2 (2) H = h(x, y) h MASK Gaussian G} (3) where is the convolution operator. 2) Gradient Magnitude: Both of the gradient magnitudes of the grey image and the reblurred image were computed. The gradients, g x,y for the gray image and h x,y for the reblurred image, are mathematically denoted as g x,y = g x,y and h x,y = h (4) x,y and the gradient magnitudes, M(g) and M(h) as shown in Fig. 3(b) and Fig. 3(d), are calculated as M(g) = gx 2 + gy 2 and M(h) = h 2 x + h 2 y (5) (c) The reblurred image (d) The gradient magnitude of reblurred image Fig. 3. Intermediate results after intentional reblur. 3) The Ratio Map: The results of the previous step were divided and the resultant ratio map is shown in Fig. 1. In (6), the numerator and the denominator are the gradient magnitudes of the gray image and the reblurred image, as described in Step 2. Step 3 is another key feature of this algorithm. The borders on the object nearer the focus plane were found to be thinner than those farther from the focus plane because the intensity transition on the edge varies in relation to the distance to the focal plane. R(g) = M(h) M(g) 4) The Focus-distance Edge Map: To compute the widths of the edges on the ratio map, a threshold was applied to convert the ratio map into a binary image, WB threshold, with a fixed threshold value of 127, as shown in (7). (6) 1, R(x, y) > threshold WB threshold (x, y) = (7) Fig. 2. The detailed flowchart of our proposed algorithm. Next, the Canny edge detector [5] was applied to obtain the edge map denoted by E. A mask as shown in (8) was then convoluted with the binary image on the edge pixels and W E (x, y), and the focus-distance edge map was produced according to (8) and (9), as shown in Fig. 1(b). On the focusdistance edge map, a smaller pixel value indicates that the pixel is closer to the focus plane. R avg (WB threshold )=WB threshold (8)

3 Fig. 4. The histogram of focus-distance edge map W E. (a) Individual priority weighting value our implementation. ing schema as an ex- (b) The weighting in (c) Another weight- W e. ample. Fig. 6. The sisteen directions to vote the values in focus map. (a) k =1 (b) k =2 (c) k =3 Fig. 5. Visualized individually for reader easier to distinctly observe D r. Fig. 7. The final full focus map after refinement with N =3. W E (x, y) = Ravg (x, y), (x, y) E 5) Extraction, Interpolation and Refinement: Finally, a histogram-slicing method was used to extract the pixels with specific ranges of gray values from the focus-distance edge image. The valleys in the histogram enclosed the ranges from v k 1 to v k. For example, Fig. 4 illustrates three slices and four cut values. The x-axis represents the bins of the divided range of values for W E and the y-axis represents the counts of pixels in W E within the specific range of values of the bin. The pixels in the same specific range fall within the same distance to the focal plane and have the same d u value. Therefore, a rough focus map of the edges was used for observation, as shown in Fig. 5. This histogram-slicing method is given in (10) and the rough focus map is denoted as D r. As we can see, the two closed flowers in front were equally distinguishable and the flower buds were in the third slicing layer. k, vk 1 <W D r (x, y) = E (x, y) <v k (10) where v i denotes the value of the dominant valleys on the histogram of W E and v i is determined by the required distinguish resolution of the generating focus map. The information of the focus map regarding non-edge pixels was estimated according to candidate results and a combination of terms, where E p denoted the neighbor edge pixels of pixel p in the 16 predefined directions, as shown in Fig. 6(a). The terms were decision strategies (judgef unc), the number of directions ( E p ), the directions of the neighbor edge pixels (E p ), the weighting for voters in various directions (W e ), and the weighting for different decision strategies (W c ). For example, W e is possibly defined as shown in Fig. 6(b). The other weighting schema shown in Fig. 6(c) is also possible. (9) Not that it is worth proposing the weighting schema for different type of scenes in the future. The rough focus map was computed by using (11) where the median, mode, mean and other decision strategies may be the judgef unc. The median and the mode were adopted in this sample implementation because they preserve the values. The resultant rough focus map is usually treated as an initial disparity map. D rough = W c judgef unc( W e D r (e)}) c judgef unc e E P (11) Then, image-segmentation methods are used to oversegment the input image into N non-overlapping, homogeneous regions, where S w represents the w th region and S i S j = φ for i j, based on a specific characteristic [6]. Other image segmentation technologies [7] also could be integrated to interpolate the partial focus map onto the complete focus map and refine the result. Within the regions, the edge pixels were the voters. The line-segment-based and region-based voting philosophies were implemented. Other decision policies can also be included and the priority-weighting mechanism can help to fit the constraints of the applications. Finally, the complete focus map was computed by using (12). The final, complete focus map is shown in Fig. 7. D full (x, y) =intpof unc(d rough (x, y) S w w s.t. (x, y) S w }) (12) Advantages of this design include the integration of the detailed and median viewpoints and the preservation of edge position and the two-dimensional characteristics of the image. The degree of quality can be determined in the configuration of N and the computational complexity can be adjusted according to the choice of judgef unc or intpof unc. A noteworthy contribution is the discovery of the ratio map,

4 which is used to extract the focus map when only a single image is available as an input. The resultant focus map is adaptive and suitable for multi-layer requirements. III. EXPERIMENTAL RESULTS AND DISCUSSION In this section, the focus maps derived from the proposed algorithm are compared with works of art. Data sets were collected from various research papers [9] [11]. The data sets Tsukuba, Venus, Cones, Undersea and Teddy are compared in the first row in Fig. 8. The second row shows that the depth map loses detailed information in the background and that the boundaries are blurry. The third row shows the results of this study. Compared to [8], the results are more accurate. For example, the books and the face are clearly visible in the Tsukuba data set. The differences in the depth map and the focus map in Fig. 8(i) and Fig. 8(n) are minor. The reason is that, in the Teddy data set, the focal plane is on the house below the bear, and the closest object in the scene is the leaf on the right. Likewise, for the Cones data set, Fig. 8(m) is compared to Fig. 8(h), where the focus is on the second row of cones. Our proposed algorithm also shows an improved result compared with the result of the graph-cut-based algorithm in the painted photo, Venus, as shown in Fig. 8(l). In the case of video, [9] used a 3D camera to capture the sequence of images and provided extra information to estimate motion and refine the result. The proposed algorithm needs only a single image and the result is superior to that of [9]. The results of this study show that the proposed approach improves the quality of focus map with human observation. Peak signal-to-noise ratio (PSNR) and mean squared error (MSE) are often used to evaluate the performance of comparable research, but are more suitable for assessing the image quality than for assessing the quality of the depth map or focus map. Similarity metrics, such as the sum of absolute intensity difference (SAD), were also not introduced because they cannot support the accuracy and the degree of detail required for a focus map. [2] S.-W. Jung and S.-J. Ko, Depth map based image enhancement using color stereopsis, Signal Processing Letters, IEEE, vol. 19, no. 5, pp , [3] W. Zhang and W.-K. Cham, Single-image refocusing and defocusing, Image Processing, IEEE Transactions on, vol. 21, no. 2, pp , [4] K. Prazdny, Egomotion and relative depth map from optical flow, Biological cybernetics, vol. 36, no. 2, pp , [5] J. Canny, A computational approach to edge detection, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. PAMI-8, no. 6, pp , [6] R. C. Gonzalez and E. Richard, Woods, digital image processing, ed: Prentice Hall Press, ISBN , [7] B. Peng, L. Zhang, and D. Zhang, A survey of graph theoretical approaches to image segmentation, Pattern Recognition, vol. 46, no. 3, pp , [8] D. Wang and K. B. Lim, Obtaining depth map from segment-based stereo matching using graph cuts, Journal of Visual Communication and Image Representation, vol. 22, no. 4, pp , [9] W.-M. Chen and S.-H. Jhang, Improving graph cuts algorithm to transform sequence of stereo image to depth map, Journal of Systems and Software, vol. 86, no. 1, pp , [10] D. Scharstein and R. Szeliski, A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, International Journal of Computer Vision, vol. 47, no. 1-3, pp. 7 42, [11], High-accuracy stereo depth maps using structured light, in Computer Vision and Pattern Recognition, Proceedings IEEE Computer Society Conference on, vol. 1, 2003, pp. I 195 I 202 vol.1. IV. CONCLUSION This article presents the spatial domain focus map estimation algorithm based on the intentional reblur of one image. The proposed framework is a hybrid of edge-based and regionbased methods and only a single image is required. This preserved the two-dimensional characteristics of the image. In other words, the edges of the original input image are usually the edges in the focus map. An amplifier operator is introduced in the proposed algorithm. This approach is valuable for two reasons: 1) It offers an efficient, precise, and improved algorithm for extracting a focus map; 2) It requires the input of only one image captured using conventional, single-view equipment without any extra hardware. REFERENCES [1] P. Favaro and S. Soatto, A variational approach to scene reconstruction and image segmentation from motion-blur cues, in Computer Vision and Pattern Recognition, CVPR Proceedings of the 2004 IEEE Computer Society Conference on, vol. 1, 2004, pp. I 631 I 637 Vol.1.

5 (a) Tsukuba (b) Venus (c) Cones (d) Teddy (e) Undersea (f) [8] (g) [8] (h) [8] (i) [8] (j) [9] (k) (l) (m) (n) (o) Fig. 8. The focus maps obtained by our proposed algorithm. Comparing to [8] and [9], the third row shows the results of this study.